Imager flicker compensation systems and methods

ABSTRACT

A method of capturing an image of a scene includes providing an imaging device having an array of pixels. Each pixel is configured to provide a pixel output. The method also includes determining a frequency of light produced by a light source illuminating the scene and capturing an image of the scene using the pixel array. A first capture time for one or more pixels in the array is different from a second capture time of one or more other pixels in the array. The method further includes, for each pixel in the array, determining a relative phase of the light with respect to the capture time of the pixel, determining a compensating gain based on the relative phase of the light and the frequency of the light, and adjusting the pixel output of the pixel in relation to the compensating gain.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application is a non-provisional of, and claims the benefit of,commonly assigned U.S. Provisional Patent Application No. 60/609,195,entitled “IMAGER FLICKER COMPENSATION,” filed on Sep. 9, 2004, nowabandoned which application is incorporated herein by reference for allpurposes.

This application is related to co-pending, commonly assigned, U.S.patent application Ser. No. 10/474,798, entitled “CMOS IMAGER FORCELLULAR APPLICATIONS AND METHODS OF USING SUCH”, filed on Oct. 8, 2003,which is a nationalization of International Patent Application No.PCT/US02/17358, entitled “CMOS IMAGER FOR CELLULAR APPLICATIONS ANDMETHODS OF USING SUCH,” filed on May 29, 2002, which application is anon-provisional of, and claims the benefit of, U.S. ProvisionalApplication No. 60/294,388, entitled “CMOS IMAGER FOR CELLULARAPPLICATIONS,” filed on May 29, 2001, the entire disclosure of each ofwhich are incorporated herein for all purposes.

BACKGROUND OF THE INVENTION

The present invention relates generally to systems and methods fordetecting and/or transmitting images. More particularly, the presentinvention relates to detecting and transmitting images in relation to acellular telephone.

Image processors typically detect light reflected off of a subject, andconvert the detected light to an electrical image. The electrical imageis then sampled at a frequency that may be the reciprocal of theintegration time. In many cases, such a frequency is different from thefrequency of light reflected off the subject. In such cases, amodulation of the displayed image can result. Such modulation appears asa flicker and can be very distracting.

To overcome this problem, some image processors allow for manualselection between different integration times that allow a user toreduce the flicker. For example, where an image processor will be usedin a 60 Hz and a 50 Hz lighting environment, a manual selection betweenan integration time associated with either 60 Hz or 50 Hz can beprovided. This alleviates some flicker, but requires adjustment from auser. Furthermore, such an approach is limited to reducing flicker in alimited number of pre-determined environments, and thus may not be ableto address various flicker situations. This is particularly problematicfor mobile devices that are used in ever changing environments.

Further, a limited reservoir of power typically exists for mobiledevices. As such, power consuming imaging applications are oftenincompatible with mobile devices. Various approaches exist to increasethe amount of power available to mobile devices, such that the powerrequirements of an imaging application are not prohibitive. However,such approaches often are both expensive and can result in increaseddimensions of a mobile device.

In some cases, light detected by an image processor is detected by apixel array and a variety of analog processing circuitry. Output fromthe pixel array representing the amount of detected light is processedby analog circuitry to produce an image which is subsequently convertedfrom the analog domain to the digital domain. The digital image issubsequently provided to a monitor of some sort for viewing.

Testing such an imaging device can include reflecting light off a knownimage, and determining if the image was properly acquired and/orprocessed by the imaging device. This method provides an effectiveapproach for testing such devices, however, the method is subject to anumber of variables including lighting and image stability which caneffect testing of the imaging device. Providing such a test, andcontrolling for such variables can be costly and time consuming.Further, such an approach tests the imaging device holistically, and islimited in its ability to identify sub-components of the imaging devicewhich may have failed.

Hence, for at least the aforementioned reasons, there exists a need inthe art to provide advanced systems, methods and devices for detectingand/or transmitting images.

BRIEF SUMMARY OF THE INVENTION

The present invention provides systems, devices, and methods forimaging. More particularly, the present invention provides devices andmethods for capturing images, formatting the images, and/or transmittingthe images via a cellular telephone network or other wireless network.

Various aspects of the present invention include an imaging device withone or more of the following units: a control unit, a serial interfaceunit, a parallel interface unit, an imaging unit, and a translationunit. In some embodiments, flicker detection and/or correctionfunctionality is incorporated. Additionally, in some embodiments, abuilt-in self test associated with various analog circuitry in theimaging device is included. Further, in some embodiments, systems andmethods for reducing power consumption by various analog circuitrywithin the imaging device are included. Yet further, in someembodiments, a parallel to serial conversion unit and associated serialoutput interface are included. These and many other novel aspects of thepresent invention are provided in greater detail in the proceedingportions of this document.

One particular embodiment of the present invention provides an imagingdevice and methods for using such that provide for advanced testingability. In one aspect, an imaging device includes a pixel array. Theimaging device further includes a selector that is operable to selectbetween an input derived from the pixel array and at least one referenceinput. Such selection between the input derived from the pixel array andthe at least one reference input is based at least in part on a signalderived from a digital domain. Further, the imaging device includes ananalog to digital converter that is operable to convert a signal derivedfrom the selector to the digital domain. In some cases, such anembodiment provides an ability to verify the functionality of analogcircuitry within the imaging device using a digital tester.

In some aspects of the embodiment, the output of the selector is passedthrough an analog processing circuit that can include one or morecircuits selected from a group consisting of: a level shift circuit, aprogrammable offset circuit, and a programmable gain amplifier. Invarious aspects of the embodiment, the imaging device further comprisesan analog processing selector that is operable to bypass one or moreportions of the analog processing circuit. In other aspects, the imagingdevice further includes a comparator receiving an output from the analogto digital converter. Such a comparator is selected from a groupconsisting of: a programmable hardware comparator, a programmablesoftware comparator, and a hardwired comparator.

In some aspects of the embodiment, the pixel array includes at least onepseudo-pixel, and the imaging device additionally includes a pixel arrayselector that is operable to select the pseudo-pixel such that inputderived from the pixel array is derived from the pseudo-pixel. Otheraspects include a column amplifier that receives and amplifies theoutput from the pseudo-pixel.

In some aspects, the reference input to the selector is a reset levelthat includes a pixel under reset condition, and the imaging devicefurther includes a circuit for selecting the reset level and pairing itwith an output from the pixel array. In this way, the selector isoperable to provide a differential pair consisting of the input derivedfrom the pixel array and the reset level. In other aspects, thereference input to the selector is tied to a pad of the imaging device,and in yet other aspects, the reference input to the selector comprisesone or more voltages generated internal to the imaging device.

In other embodiments of the present invention, imaging devices andmethods for using such are provided to detect and/or reduce imageflicker. Such an imaging device includes an image sensor with one ormore rows of pixels, where each of the pixels provides a pixel value.The device further includes a storage element and a summing circuit.Such a summing circuit can be a hardware summing circuit, eitherprogrammable or hardwired, or a software summing circuit including forexample a programmable core. The summing circuit is operable to sum thepixel values for each of the one or more rows to create row sums, and tostore the row sums to the storage element. In addition, the imagingdevice includes a programmable core that includes instructionsexecutable by the programmable core to: determine a first energy valueand a second energy value associated with the row sums; compare thefirst energy value and the second energy value to estimate a flickerfrequency; and constrain an exposure duration associated with the imagesensor to approximate a multiple of the flicker frequency. Such aprogrammable core can also be used for the summing circuit.

In some aspects of the embodiment, the summing circuit is furtheroperable to calculate row sums for a plurality of frames, where each ofthe plurality of frames includes pixel values from one or more rows ofpixels, and where the instructions are further executable by theprogrammable core to: determine a first energy value and a second energyvalue associated with the row sums for each frame; calculate a variancebetween first energy values across the plurality of frames; calculate avariance between second energy values across the plurality of frames;and wherein the comparing the first energy value and the second energyvalue to estimate a flicker frequency comprises comparing the varianceof the first energy values with the variance of the second energyvalues. In yet other aspects of the embodiment, the instructions arefurther executable by the programmable core to: compare the variance ofthe first energy values with the variance of the second energy values,and to compare the greater variance with a predefined threshold whereconstraining the exposure frequency is done when the greater variance isgreater than the predefined threshold.

In various aspects, a confidence counter is implemented to controlswitching from an exposure frequency to a multiple of an estimatedflicker frequency. In such aspects, a change of frequencies can belimited to situations where the confidence counter changes sign.

In one particular aspect of the embodiment, the summing circuit isfurther operable to calculate row sums for a plurality of frames,wherein each of the plurality of frames includes pixel values from thetwo or more rows of pixels. Further, the imaging device further includesan interrupt associated with the summing circuit, wherein assertion ofthe interrupt indicates that summing for the two or more rows of pixelsfor a frame is complete; and wherein the instructions are furtherexecutable by the programmable core to: receive the interrupt; determinea first energy value and a second energy value associated with the rowsums for each frame; determine if a sufficient number of frames havebeen received to calculate a variance between first energy values and avariance between second energy values; calculate the variance betweenfirst energy values across the plurality of frames; calculate thevariance between second energy values across the plurality of frames;and wherein the comparing the first energy value and the second energyvalue to estimate a flicker frequency comprises comparing the varianceof the first energy values with the variance of the second energyvalues.

Some aspects of the invention provide a method for automaticallydetecting flicker in an imaging device. The method includes providing animaging device, wherein the imaging device includes an image sensor withtwo or more rows of pixels, and wherein each of the pixels provides apixel value. In addition, the method includes retrieving at least aportion of the two or more rows of pixels, and summing the pixel valuesto create a row sum for each of the rows of pixels; and determining afirst energy value and a second energy value associated with the rowsums. The first and second energy values are compared to estimate aflicker frequency.

Various embodiments of the present invention include a pixel with acharge evacuation mechanism for reducing noise evident at the output ofthe pixel. In one aspect, the pixel includes a reset element used inrelation to charging a light sensitive element. The evacuation mechanismis used to dissipate charge build up about the reset element.Dissipation of the charge results in a reduction of noise at the outputof the pixel. In particular embodiments, such a pixel and/or chargeevacuation mechanism is implemented using complementary metal oxidesemiconductor (“CMOS”), field effect transistor (“FET”) technology.

Further, aspects of the present invention include systems and methodsfor utilizing such pixels. More particularly, such methods includeapproaches for applying various control signals to the pixel and systemsinclude elements for producing the various control signals.

One particular aspect of the invention provides a MOS pixel thatincludes a reset transistor with a reset drain and a reset gate.Further, the pixel includes a charge evacuation element that iselectrically coupled to the reset drain and is operable to evacuatecharge accumulated in a channel of the reset transistor. In someembodiments, the charge evacuation element is a MOS transistor with acharge evacuation drain, a charge evacuation source, and a chargeevacuation gate. In particular cases, the charge evacuation source andthe charge evacuation drain are both electrically coupled to the resetdrain. In various embodiments, the pixel further includes a lightsensitive element electrically coupled to the reset drain.

Further embodiments comprise a source follower transistor that includesa source follower gate and a source follower drain, the source followergate being electrically coupled to the reset drain. Such embodimentsfurther include a selection transistor with a selection drain, aselection source, and a selection gate, where the selection source iselectrically coupled to the source follower drain. A selection signal iselectrically coupled to the source gate such that assertion thereofcauses a representation of a signal from the node of the light sensitiveelement to be present on an output of the pixel.

In some embodiments, the charge evacuation element is a MOS transistorwith a charge evacuation source and a charge evacuation drain, both ofwhich are coupled to the reset drain. The charge evacuation elementfurther includes a charge evacuation gate. Further, such embodiments caninclude a reset signal driving the reset gate and a complement of thereset signal driving the charge evacuation gate. In some instances, thecomplement of the reset signal is delayed from the reset signal.

Other embodiments of the present invention provide a circuit including areset transistor comprising a reset gate and a reset drain. The circuitfurther includes a photodiode with a node of the photodiode electricallycoupled to the reset drain, and a charge evacuation transistor thatincludes a charge evacuation gate, a charge evacuation drain, and acharge evacuation source. The charge evacuation source and the chargeevacuation drain are both electrically coupled to the reset drain andthe charge evacuation transistor is operable to evacuate chargeaccumulated in a channel of the reset transistor.

Various embodiments of the circuit also include a source followertransistor comprised of a source follower gate and a source followerdrain. The source follower gate is electrically coupled to the resetdrain. the circuit further includes a selection transistor comprised ofa selection source, a selection gate, and a selection drain. Theselection source is electrically coupled to the source follower drain. Aselection signal is provided that when asserted causes a representationof a signal from the photodiode to be present on an output of the pixel.

Yet other embodiments of the present invention provide an imagingsystem. Such an imaging system includes a group of pixel elements. Oneor more of the pixel elements includes a reset element, a chargeevacuation element, and a light sensitive element. In some cases, theimaging system further includes an optical device, wherein the opticaldevice transfers light to the group of pixel elements, and wherein thelight strikes the light sensitive element of the pixel elements. Inaddition, a timing circuit is included that provides at least onecontrol signal to the group of pixel elements. In one particularinstance, the group of pixel elements are arranged in a rectangulararray.

Yet another embodiment of the present invention provides an image sensorincluding a plurality of pixel devices. The plurality of pixel devicesare arranged as a plurality of rows and a plurality of columns. Each ofthe pixel devices includes: a light detecting element, and a chargeevacuation element for dissipating unwanted charge built up in the imagesensor.

Yet additional embodiments provide methods for detecting an image. Suchmethods include providing a pixel device that comprises a chargeevacuation element, and applying a charge evacuation control signal tothe charge evacuation element, wherein a charge accumulation in achannel of a reset transistor of the pixel device is reduced.

Other embodiments provide systems and method for reducing powerconsumption in an imaging device. In one aspect, an imaging device isincluded with an image sensor and a sub-sample control system. Thesub-sample control system is operable to identify one or more signalsfrom the image sensor that are not processed. The imaging device furtherincludes an analog processing circuit that receives the one or moresignals from the image sensor at one or more inputs, and a processingcontrol system that is operable to limit switching of the one or moreinputs. In some aspects, the analog processing circuit comprises acolumn amplifier that receives the one or more inputs. In variousaspects, the analog processing circuit comprises a level shift circuit,and/or a programmable or black offset circuit.

In some aspects, the processing control system is operable to maintain acommon mode feedback loop of an amplifier associated with the analogprocessing circuit in an active state, while limiting switching of theone or more inputs. In other aspects, the imaging device furtherincludes a digital processing circuit, and the processing control systemis further operable to disable the digital processing circuit.

Yet other embodiments of the present invention provide a CMOS imagerthat includes an image sensor, and a parallel to serial data conversionunit. The parallel to serial data conversion unit receives an imagederived from the image sensor and converts the image to a serial datastream. The serial data stream is output via a serial output interface.In some aspects of the embodiment, the image sensor comprises one ormore pixels, and wherein the one or more pixels include a chargeevacuation element. in other aspects, the serial output interfaceincludes a clock signal and a data signal. The clock signal is onlyactive when valid data is presented on the data signal. In yet otheraspects, the serial output includes a clock signal, a data signal, and aqualifying signal. The qualifying signal indicates the presence of validdata on the data signal.

Still other embodiments provide a method of capturing an image of ascene. The method includes providing an imaging device having an arrayof pixels. Each pixel is configured to provide a pixel output. Themethod also includes determining a frequency of light produced by alight source illuminating the scene and capturing an image of the sceneusing the pixel array. A first capture time for one or more pixels inthe array is different from a second capture time of one or more otherpixels in the array. The method further includes, for each pixel in thearray, determining a relative phase of the light with respect to thecapture time of the pixel, determining a compensating gain based on therelative phase of the light and the frequency of the light, andadjusting the pixel output of the pixel in relation to the compensatinggain. The imaging device may be a CMOS imaging device. Determining afrequency of light produced by a light source illuminating the scene mayinclude capturing an image of a generally uniform scene and determininga variation between pixel outputs for one or more pixels in the array ofpixels. Determining a compensating gain based on the relative phase ofthe light and the frequency of the light also may include determining acompensating gain based on a color associated with the pixel.Determining a frequency of light produced by a light source illuminatingthe scene may include calculating a sum of pixel outputs for a pluralityof pixels in a row of pixels and spectrally analyzing the sum.Calculating at least two energy values associated with the sum mayinclude comparing the at least two energy values.

In still other embodiments, an image capture device includes an array ofpixels. Each pixel is configured to provide a pixel output. The devicealso includes a programmable core. The programmable core includesinstructions executable by the programmable core to determine afrequency of light produced by a light source illuminating a scene andcause the pixel array to capture an image of the scene. A first capturetime for one or more pixels in the array is different from a secondcapture time of one or more other pixels in the array. The core alsoincludes instructions executable by the programmable core to, for eachpixel in the array, determine a relative phase of the light with respectto the capture time of the pixel, determine a compensating gain based onthe relative phase of the light and the frequency of the light, andadjust the pixel output of the pixel in relation to the compensatinggain. The image capture device may be a CMOS image capture device. Inprogramming the core to determine a frequency of light produced by alight source illuminating the scene, the instructions may program thecore to capture an image of a generally uniform scene and determine avariation between pixel outputs for one or more pixels in the array ofpixels. In programming the core to determining a compensating gain basedon the relative phase of the light and the frequency of the light, theinstructions may program the core to determine a compensating gain basedon a color associated with the pixel. In programming the core todetermine a frequency of light produced by a light source illuminatingthe scene, the instructions may program the core to calculate a sum ofpixel outputs for a plurality of pixels in a row of pixels andspectrally analyze the sum. In programming the core to determine afrequency of light produced by a light source illuminating the scene,the instructions may further program the core to calculate at least twoenergy values associated with the sum and compare the at least twoenergy values.

In still other embodiments, an image capture device includes an array ofpixels. Each pixel is configured to provide a pixel output. The devicealso includes means for determining a frequency of light produced by alight source illuminating a scene, means for determining a phase of thelight relative to a pixel capture time, means for determining acompensating gain for each pixel in the array based on the frequency andthe phase, and means for adjusting each pixel's pixel output, during animage capture, in relation to the pixel's compensating gain. The imagecapture device may be a CMOS image capture device. The means fordetermining a frequency of light produced by a light source illuminatingthe scene may include means for capturing an image of a generallyuniform scene and means for determining a variation between pixeloutputs for one or more pixels in the array of pixels. Each pixel of thearray may include a plurality of sub-pixels, in which case the imagecapture device may include means for determining a compensating gain foreach sub-pixel of each pixel. The means for determining a frequency oflight produced by a light source illuminating the scene may includemeans for calculating a sum of pixel outputs for a plurality of pixelsin a row of pixels and means for spectrally analyzing the sum. The meansfor determining a frequency of light produced by a light sourceilluminating the scene may include means for calculating at least twoenergy values associated with the sum and means for comparing the atleast two energy values.

These and other aspects are more fully developed in the detaileddescription below. Thus, the summary provides only a general outline ofthe embodiments according to the present invention. Many other objects,features and advantages of the present invention will become more fullyapparent from the following detailed description, the appended claimsand the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

A further understanding of the nature and advantages of the presentinvention may be realized by reference to the figures which aredescribed in remaining portions of the specification. In the figures,like reference numerals are used throughout several figures to refer tosimilar components. In some instances, a sub-label consisting of a lowercase letter is associated with a reference numeral to denote one ofmultiple similar components. When reference is made to a referencenumeral without specification to an existing sub-label, it is intendedto refer to all such multiple similar components.

FIG. 1 is a block diagram of a CMOS imager in accordance with variousembodiments of the present invention;

FIG. 2 is a timing diagram of frame rate control implemented in the CMOSimager of FIG. 1;

FIG. 3 is a timing diagram illustrating the various clocks used inrelation to the CMOS imager of FIG. 1;

FIG. 4 is a block diagram of a sensor in accordance with one embodimentof the present invention;

FIG. 5 is a block diagram of a virtual image window in accordance withvarious embodiments of the present invention;

FIG. 6 illustrates a Bayer grid array;

FIG. 7 illustrates a pixel circuit in accordance with variousembodiments of the present invention;

FIG. 8 is a flow diagram illustrating a method for using the pixelcircuit of FIG. 7 in accordance with an embodiment of the presentinvention;

FIGS. 9 are timing diagrams illustrating the operation of the pixel ofFIG. 7.

FIG. 10 is an exemplary column amplifier circuit;

FIGS. 11-13 are timing diagrams illustrating the operation of the columnamplifier circuit of FIG. 10;

FIG. 14 is a timing diagram illustrating the exposure and sensing of aframe of video in accordance with an embodiment of the presentinvention;

FIG. 15 is a timing diagram illustrating sampling of a pixel arrayincorporated in the sensor of FIG. 4;

FIG. 16 illustrates high gain and high dynamic range modes of the levelshift circuit of FIG. 4;

FIG. 17 illustrates the use of black level and gain controls in relationto the sensor of FIG. 4;

FIG. 18 is a timing diagram of various control signals associated withthe sensor of FIG. 4;

FIG. 19 is a timing diagram illustrating the interaction of variousstandby mode control signals;

FIG. 20 is a block diagram of a standby mode circuit in accordance withembodiments of the present invention;

FIGS. 21 illustrates an implementation of a standby mode in a switchedcapacitor circuit in accordance with embodiments of the presentinvention;

FIG. 22 is a flow diagram illustrating a method of entering and exitinga standby mode in accordance with embodiments of the present invention;

FIG. 23 is a block diagram of a power distribution circuit for use inrelation to the CMOS imager of FIG. 1;

FIG. 24 is a block diagram of a translation unit for use in relation tothe CMOS imager of FIG. 1;

FIG. 25 illustrates the placement of color components by the translationunit of FIG. 24;

FIG. 26 is a block diagram of a flicker detection and/or correctionblock in accordance with embodiments of the present invention;

FIG. 27 is a graph illustrating an exemplary row average across a frameof data;

FIG. 28 is a flow diagram of a first method for detecting and/orcorrecting flicker in accordance with embodiments of the presentinvention;

FIG. 29 is a flow diagram of a second method for detecting and/orcorrecting flicker in accordance with embodiments of the presentinvention;

FIG. 30 is a timing/block diagram illustrating the operation ofqualification signals used in relation to providing parallel output datafrom the CMOS imager of FIG. 1;

FIG. 31 is a timing diagram illustrating vertical and horizontalqualification signals used in relation to providing parallel output datafrom the CMOS imager of FIG. 1;

FIG. 32 is a timing diagram illustrating the output of Bayer data via aparallel output interface;

FIG. 33 is a timing diagram illustrating the output of YUV data via aparallel output interface;

FIG. 34 is a timing diagram illustrating frame dropping;

FIG. 35 is a timing diagram illustrating unqualified serial output inaccordance with embodiments of the present invention; and

FIG. 36 is a timing diagram illustrating qualified serial output inaccordance with embodiments of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

The present invention provides systems, devices, and methods forimaging. More particularly, the present invention provides devices andmethods for capturing images, formatting the images, and/or transmittingthe images via a cellular telephone network or other wireless network.

In one particular embodiment of the present invention, an imaging deviceincluding a CMOS sensor is provided. The imaging device includes animager array, sensor timing control, and image processing algorithmsintegrated onto a single chip. Such an imaging device can produce CIF(i.e., 352×288) resolution image for still capture, and QCIF (i.e.,176×144) resolution image for video applications. In addition, theimaging device can perform various correction algorithms on a capturedimage to create a corrected YCrCb (4:2:2) image.

In an embodiment, the imager array is 384×316 physical pixels including376×296 active pixels. The pixels can be mapped to a configurable sizeof output Bayer grid array (e.g., windowing). Pixel size can be 6.4μm×6.4 μm. QCIF resolution can be created by either or both ofdown-sampling and filtering.

In some embodiments, a resulting Bayer grid (raw RGB output) or arecreated YCrCb image is sent out on a digital ten (10) bit parallelinterface. The ten (10) bit parallel interface provides access to acaptured video image on a line by line basis. In addition, an outputclock and other qualifying signals are provided in association with theten (10) bit parallel interface.

The imaging device can include electronic exposure control from 375 nsecup to a full frame (e.g., 33.33 msec at a maximum frame rate of thirtyframes per second). Further, automatic dark level offset, a separategain for each of the color components, and configurable offset fordynamic range stretching, as well as, image enhancement algorithms canbe provided. Such image enhancement algorithms can include black leveloffset, fault pixel correction, color interpolation, color and whitebalance correction, gamma correction, and/or auto exposure and autowhite balance algorithms. In addition, noise reduction processes can beapplied to reduce noise in general, and fixed pattern noise (“FPN”) inparticular.

The imaging device can operate on a single 2.8 volt power supply and areduced-power mode can be implemented to reduce power consumption whenfull operation of the imaging device is not necessary. In addition, theimaging device can include a sleep mode for further reducing powerconsumption.

In some embodiments, the imaging device is designed for use under thecontrol of an external processor, which in some cases is a DigitalSignal Processor. The external processor can act as a system host(“Host”). As such, the external processor controls the mode of theimaging device, monitors system status, and compresses an output imagestream. For example, the external processor can control the frame-rate,the size of the requested image, and several registers for imageprocessing, such as gamma-correction and gain registers. Such controlcan be provided via a programmable core integrated with the imagingdevice.

Block Diagram

FIG. 1 illustrates a block diagram of an imaging device 100 inaccordance with embodiments of the present invention. Imaging device 100includes a serial interface unit 106, a control unit 110, an imagingunit 130, a translation unit 150, a testability unit 170, and a parallelinterface unit 190. The various units are electrically coupled via acontrol bus 101, an image bus 131, a translation bus 151, a parallelinterface bus 191, a UART bus 120, a DeICE bus 125, and a serial bus107. Inputs and outputs to/from imaging device 100 can be provided viaan I2C input/output (“I/O”) 108, a UART I/O 121, a DeICE I/O 126, and/ora parallel video interface 192. In addition, imaging device 100 includesa clock unit 102 with a reset 171 and a clock 172 input.

Serial Interface

In one embodiment, serial I/O 108 includes a clock and data input. Sucha serial interface can be used to program various registers of imagingdevice 100. Various different serial interface types that are known inthe art can be incorporated into imaging device 100 in accordance withthe present invention.

Control Unit

Control unit 110 implements parameters used in relation to the differentunits within imaging device 100, and calculates various parameters forimage enhancement. In some embodiments, control unit 110 includes aprogrammable processor core 112, and in one particular embodiment,programmable processor core 112 is a 80186Turbo processor core. Controlunit 110 also includes a core memory 118, and interface hardware (notshown) to UART I/O 121 and DeICE 126 I/O. Further, control unit 110includes an arbitrator 116 for controlling access to control bus 101,and a frame-rate mechanism 114 for controlling the frame rate outputvideo images from imaging device 100.

Via arbitrator 116, programmable processing core 112, serial I/O 108,UART I/O 121, and/or DeICE I/O 126 can access registers maintained onimaging device 100 over control bus 101. In one particular embodiment,when programmable processing core 112 is disabled by, for example,entering a clock disable mode, serial I/O 108 is granted immediateaccess to control bus 101 without adhering to an arbitration schemeimplemented by arbitrator 116. Such a clock disable mode is discussed inmore detail with reference to clock unit 102. For purposes of theimmediate discussion, clock unit 102 produces a control line thatindicates if programmable processing core 112 is active, or if it isheld in reset mode or has its clock shut down. Alternatively, whenprogrammable processing core 112 is active, arbitrator 116 grants accessto control bus 101 according to a determined priority scheme. One suchpriority scheme is described in the 80186Turbo reference manual.

Programmable processor core 112 serves as the main control source forimaging device 100, controlling other units via control bus 101. Suchcontrol can be provided by manipulating address, data and read/writesignals associated with control bus 101. As just one example,programmable processing core 112 sets up the working mode of imagingdevice 100 according to the parameters that are received from a host(not shown) via serial I/O 108, UART I/O 121, and/or DeICE I/O 126. Inaddition, programmable processing core 112 can regulate the operation ofthe various units within imaging device 100.

In one particular embodiment, programmable processor core 112 supports atwenty-bit address memory space for data and program memories, and afourteen-bit address spaced for I/O mapped peripherals. In otherembodiments, only a twelve-bit memory space and an eight-bit I/O space(program and data) are supported.

Programs controlling the operation of imaging device 100 can be loadedto core memory 118 via serial I/O 108, and/or via UART I/O 121. Once theprogram is loaded, control unit 110 calculates and updates various imageprocessing parameters according to a selected working mode. After acomplete frame is read from imaging unit 130 and processed bytranslation unit 150, control unit 110 reads statistical results of theframe, and produces the exposure, gain and auto-white balance parametersfor the next frames. In this way, control unit 110 continuously monitorsoperation of imaging device 100 and modifies such operation to assurethat it conforms to the downloaded program.

In one particular embodiment, various parameters associated with theoperation of imaging device 100 are written to registers associated withthe various units within imaging device 100. Such a process can beperformed by writing the parameters to core memory 118, after whichprogrammable processing core 112 writes the values to registersmaintained within the various units to which the parameters pertain.Alternatively, such parameters can be written directly to the variousunits by the Host. To facilitate such processes, imaging unit 130,translation unit 150, and parallel interface unit 190 can include doublebuffered registers so that parameters can be loaded into such registersat any time without interfering with continuing processes.

Imaging device 100 operates in a default standby mode, with sensor 132turned off and translation unit 150 and parallel interface unit 190inactive. Once the program is loaded, control unit 110 awaitsinstructions from the Host. The Host can pass commands to control unit110 by writing to memory 118 at HOST-FIFO address space. After writingdata to the HOST-FIFO address space, the Host can activate an interruptto programmable processing core 112. Programmable processing core 112then reads the data and responds according to the instructions.

Once the frame counters are active, a vertical synchronization interruptsignal (“VSYNC”) is asserted every time a frame boundary for an image isdetected. VSYNC is used by programmable processing core 112 to governsoftware frame-rate control, or by frame-rate mechanism 114 to governhardware frame-rate control. When an image is being processed by imagingdevice 100, statistics are collected in translation unit 150. At the endof a frame processing, translation unit 150 activates an interruptsignal, END_TRANSLATION, indicating that programmable processing core112 can read the statistics from translation unit 150. Programmableprocessing core 112 utilizes these statistics to calculate exposure,gain and color processing parameters for the next frames. In someembodiments, such calculations are provided in accordance withauto-exposure, auto-white-balance, and/or dynamic range algorithms. Thecalculations result in a new set of parameters that are written bycontrol unit 110 to registers maintained in association with imagingunit 130 and translation unit 150 before the following VSYNC.

Further, control unit 110 can be interrupted by either or both of UARTI/O 121 and serial I/O 108. Such interrupts are generally processed assoon as programmable processing core 112 becomes available. Reasonableresponse time to such interrupts helps assure that the I/O pipelines donot become overfilled. The following Table 1 lists a series of taskscompleted as imaging device 100 is programmed and started.

TABLE 1 Imaging Device Tasks EVENT Task Comments Reset Programmableprocessing core 112 is frozen Deassert Reset Execute boot program -check configuration register. If boot from UART I/O 121, wait for UARTmessage. If from serial I/O 108, program is loaded, and can subsequentlybe executed. According to program, initiate registers and activateimager frame counters. VSYNC Asserts Interrupt processes. Activatetranslation unit 150 First frame exposure has for next frame. Executestarted, no image is being frame-rate control routine. read out. VSYNCInterrupt processes. Asserts Execute frame-rate control Image is beingsent from routine imaging unit 130 to translation unit 150 in accordancewith a frame-rate control. END_TRANSLATION Interrupt processes. AssertsRead statistics results from The algorithm defines the translation unit150. sequence of the calculations. Calculate gain and offset for In someinstances, not all imaging unit 130. Calculate calculations areperformed exposure parameters, white for every frame. balancecorrection, and color correction matrix. Write calculated parameters tovarious units. VSYNC Asserts Interrupt processes. Execute frame-ratecontrol routine.

In one embodiment, programmable processing core 112 includes sixinterrupts. Such interrupts can include, but are not limited to, twosoftware timer interrupts used for timeout purposes, and interruptsassociated with UART I/O 121 and serial I/O 108. Additionally, VSYNC andEND_TRANSLATION are provided. As previously discussed, VSYNC indicates aframe boundary has been detected, and END_TRANSLATION indicates thattranslation unit 150 has completed processing a frame. Upon assertion ofEND_TRANSLATION, translation has gathered and/or calculated variousstatistics associated with the processed frame that can now be read andutilized by programmable processing core 112.

Programmable processing core 112 utilizes the statistics to perform oneor more of the following algorithms: auto exposure (e.g., exposureparameters calculation), auto white balance (e.g., white balancecoefficients calculation and/or full color matrix calculation), anddynamic range stretching, (e.g., gain and offset calculations). Thus,for example, if the auto-exposure and auto-white-Balance options areenabled, the parameters calculated by programmable processing core 112in relation to such controls are passed to imaging unit 130 where theyare used to control exposure and processing of subsequent image frames.

In addition, programmable processing core 112 can turn its own clock offby writing to a disable clock register maintained in clock unit 102. Insome embodiments, the clock is turned back on whenever any activity isdetected on a receive pin associated with UART I/O 121, or if anyinterrupt signal is activated. Disabling programmable processing core112 helps to conserve power.

In addition to controlling the rate of image output via parallel videointerface 192, control unit 110 also controls the frame-rate associatedwith image processing internal to imaging device 100. In an embodiment,three registers are used to effectuate frame rate control. The first,Frame_rate_en, holds an enable bit for a frame-rate mechanism 114. Whenframe-rate mechanism 114 is disabled, the array content is readaccording to a second register, Frame_rate_read, and the data is sentout on parallel interface 192. When frame-rate mechanism 114 isactivated, two eight-bit registers define whether a particular frame isto be read out, or dropped. When a frame is to be output via parallelvideo interface 192, a control signal, c_readframe, which issynchronized to a frame boundary signal, Im_Frame 910, is sent toimaging unit 130 to enable output of the frame data to translation unit150. In addition, a control signal, C_Outframe, which is alsosynchronized to Im_Frame 910 signal is sent to parallel interface unit190 to enable the data received from translation unit 150 to be outputvia parallel video interface 192 at a rate defined by theFrame_rate_send register value.

As illustrated in FIG. 2, when c_readframe is asserted high, the contentof a pixel array within sensor 132 is transferred from imaging unit 130to translation unit 150. When C_Outframe is asserted high, the data issent out on parallel video interface 192. In some embodiments, whenC_Outframe is deasserted, the output data is ignored by parallelinterface unit 190, but the output pins are driven. While the data linesof parallel video interface 192 are driven, the vertical and horizontalsynchronization signals associated with the data are not driven. Inother embodiments, when C_Outframe is deasserted, the output data isignored by parallel interface unit 190, and the output pins are notdriven to conserve power and reduce noise.

Clock Unit

Clock unit 102 receives clock input 172 signal and reset input 171.Based on these signals, as well as control signals from control unit110, clock unit 102 produces the various clock and reset signals to thedifferent units within imaging device 100. In addition, clock unit 102samples the configuration pins while reset signal 171 is asserted, andproduces several configuration signals based upon the sensedconfiguration arrangement. The configuration inputs determine theworking clock rate, the type of serial interface (SSI or I2C compatible)that is operational on serial I/O 108 and the program-loading source(serial interface or UART) that is operational on UART I/O 121. Further,clock unit 102 produces a control line that indicates if programmableprocessing core 112 is active, or if it is held in reset mode or has itsclock shut down. As previously discussed, this signal enables the bypassof arbitrator 116 for accessing control bus 101.

Clock unit 102 produces the device's general reset signal. An internalreset signal, Reset_i 104, is activated when reset input 171 isasserted, by the setting of a software reset register, or by change inphase locked loop configuration. Clock unit 102 also synchronizesReset_i 104 to an internal clock signal, Rclk 103.

Clock unit 102 also produces a signal that resets programmableprocessing core 112. This reset signal is activated when Reset_i 104 isactive and/or by a special reset command from serial interface unit 106.When the configuration of imaging device 100 calls for program loadingthrough serial interface unit 106, programmable processing core 112 isheld in reset state by this signal until released by a load completecommand.

For power reduction purposes, in some embodiments clock signals fromclock unit 102 may be turned off. This can be accomplished underdirection from either programmable processing core 112 or the Host. Aspreviously discussed, the clocks can be restored whenever an interruptsignal is detected. Alternatively, clocks going to units other thancontrol unit 110 can be turned off, and subsequently restored by issuinga command from programmable processing core 112.

In an embodiment, clock unit 102 produces the following signals: an Rclk601, an Aclk 603, a Reg_clk 609, an Rc_pix signal 605, an Rclkc, anRclky, and an Rclkp. Rclk 601 is the main internal clock signalgenerated from clock input 172. A division factor is applied to clockinput 172 to produce Rclk 601, such division factor being determined byreading the configuration pins when reset input 171 is asserted. Aclk603 is a clock signal to the analog part of imaging unit 130 and issynchronous to Rclk 601, with a one-half Rclk 601 cycle skew betweenthem, at the rate of the pixel-cycles. Thus, the positive going edge ofAclk 603 follows every fourth positive edge of Rclk 601.

Reg_clk 609 is a clock signal to the voltage regulators. It isone-sixteenth the rate of Rclk 601. The rising edge of Reg_clk 609coincides with the rising edge of Rclk 601, when Rc_pix 605 is asserted.Rc_pix 605 is a signal in the digital domain that marks the Rclk 601cycle that coincides with the first low cycle of Aclk 603. The firstpositive going clock edge after an indication that a phase lock loop isready is qualified as Rc_pix 605. Rclkc is a clock signal to controlunit 110 that is identical to Rclk 601. Similarly, Rclky is identical toRclk 601 and is the clock signal to translation unit 150. Rclkp is aclock signal to parallel interface 190 that is derived from clock input172. Rclkp can either be identical in frequency to clock input 172, ordivided according to a clock division register, Rclkp_config. Rclkp isused for serial output mode and is typically eight to sixteen times therate of a pixel-clock when serial output is required. Rclk_edge 607 is aqualifying signal to parallel interface unit 190 and is used for serialoutput mode. FIG. 3 is an exemplary timing diagram 600 illustrating oneexample of the timing of the various signals generated by clock unit102.

The operating mode of imaging device 100 determines the clockconfiguration. The following Table 2 lists some possible operatingmodes, and the clocking scheme that is associated with each of themodes:

TABLE 2 Clock Modes Input Sup- Rclk Rclkp fre- ported divide dividequency fps Supported output format factor factor  8 MHz 15 fps Bayer,YUV parallel (CIF, QCIF) 1 NA Bayer 1x serial - QCIF 1 1 12 MHz 20 fpsBayer, YUV parallel (CIF, QCIF) 1 NA Bayer 1x serial - QCIF 1 1 16 MHz30 fps Bayer, YUV parallel (CIF, QCIF) 1 NA Bayer 1x serial - QCIF 1 115 fps Bayer, YUV parallel (CIF, QCIF) 2 NA Bayer 1x serial - QCIF 2 2Bayer 1x serial - CIF 2 1 YUV 1x serial - QCIF 2 1 Bayer 2x serial -QCIF 2 1 32 MHz 30 fps Bayer, YUV parallel (CIF, QCIF) 2 NA Bayer 1xserial - QCIF 2 2 Bayer 1x serial - CIF 2 1 YUV 1x serial - QCIF 2 1Bayer 2x serial - QCIF 2 1 15 fps Bayer, YUV parallel (CIF, QCIF) 4 NABayer 1x serial - QCIF 4 4 Bayer 1x serial - CIF 4 2 YUV 1x serial -QCIF 4 2 YUV 1x serial - CIF 4 1 Bayer 2x serial - QCIF 4 2 Bayer 2xserial - CIF 4 1 YUV 2x serial - QCIF 4 1 64 MHz 30 fps Bayer, YUVparallel (CIF, QCIF) 4 NA Bayer 1x serial - QCIF 4 4 Bayer 1x serial -CIF 4 2 YUV 1x serial - QCIF 4 2 YUV 1x serial - CIF 4 1 Bayer 2xserial - QCIF 4 2 Bayer 2x serial - CIF 4 1 YUV 2x serial - QCIF 4 1 15fps Bayer, YUV parallel (CIF, QCIF) 8 NA Bayer 1x serial - QCIF 8 8Bayer 1x serial - CIF 8 4 YUV 1x serial - QCIF 8 4 YUV 1x serial - CIF 82 Bayer 2x serial - QCIF 8 4 Bayer 2x serial - CIF 8 2 YUV 2x serial -QCIF 8 2 YUV 2x serial - CIF 8 1

Various combinations of frequencies for the input clock, Rclk and Rclkpare summarized in Table 3 below:

TABLE 3 Various Input and Output Clock Frequencies clock input 172 Rclkdividing factor Rclkp dividing factor) 8 MHz, 12 MHz, 16 MHz 1 1 16 MHz,32 MHz 2 1 2 2 32 MHz, 64 MHz 4 1 4 2 4 4 64 MHz 8 1 8 2 8 4 8 8

Imaging Unit

Imaging unit 130 provides the central functionality of imaging device100. Imaging unit 130 includes a timing control and address generatorunit (“TCAG”) 136 that receives timing signals from control unit 110 andproduces the sequence signals to sensor 132 and ADC 134 for integration(e.g., exposure control), read and reset operations. Sensor 132 includesa CIF size pixel array, level shift circuitry, gain control circuitry,programmable or black offset circuitry, gamma-correction circuitry andreadout circuits.

FIG. 4 illustrates a block diagram 792 of the analog processing portionof imaging unit 130 including sensor 132 and ADC 134. Sensor 132includes a pixel array 700, a column amplifier 703, an analogmultiplexer (“MUX”) 709, a source selector mux 713, a level shift block719, a black offset block 723, a first stage programmable gain amplifier729, and a second stage programmable gain amplifier 733. In addition,sensor 132 includes a column logic block 739 and a reset level block743. It should be noted that column amplifier 703 represents a vector ofcolumn amplifiers. Column logic within imaging device 100 can bedesigned such that only two of the 384 total column amplifiers withincolumn amplifier 703 are active at any given time.

FIG. 5, illustrates one embodiment of pixel array 700 associated withsensor 132. Pixel array 700 includes an array of 376 by 296 activepixels 720 surrounded by four black columns 730 on each side. It is alsosurrounded by sixteen black lines 710 above and four black lines belowthe active image array, totaling 384 by 316. Each pixel in pixel array700 is covered by a color filter, thus creating a Bayer grid format 800as illustrated in FIG. 6.

Referring again to FIG. 4, sensor 132 receives a number of timing andaddress signals from TCAG 136 (FIG. 1). More particularly, pixel array700 receives the decoded and time-shaped line reset, Im_line_rst[328:0](element 707), and line select, Im_line_sel[328:0] (element 727),signals.

In some embodiments of imaging device 100, each pixel within pixel array700 includes a charge injection compensation circuit. Referring to FIG.7, a pixel 3000 is illustrated in accordance with embodiments of thepresent invention that include such a charge injection compensationcircuit. Pixel 3000 includes a reset element 3350, a source followertransistor 3360, a selection element 3370, a charge evacuation element3380, and a light sensitive element 3390. The inputs to pixel 3000include a reset control signal (im_line_rst 707), a selection controlsignal im_line_sel 727), and a charge evacuation control signal(im_line_rst_n 717). Further, pixel 3000 provides an output signal 742.In an embodiment, pixel 3000 is attached to a voltage source 3345 and acommon ground 3355.

Source follower transistor 3360 can be any of a number of transistortypes. In one particular embodiment, source follower transistor 3360 isa CMOS FET. Source follower transistor 3360 includes a source followergate 3361, a source follower source 3362 and a source follower drain3363. Each of evacuation element 3380, reset element 3350, andphotodiode 3390 are electrically connected to source follower gate 3361.For purposes of this document, charge evacuation element 3380 can be anydevice capable of dissipating charge from pixel 3000. Thus, for example,charge evacuation element 3380 can be a switch, a FET, a bipolarjunction transistor (“BJT”), a switched capacitor, or any other suchdevice. In one particular embodiment of the present invention, chargeevacuation element 3380 is CMOS transistor as illustrated in FIG. 7. Insome cases, such CMOS transistors can be comprised of N-FET devices,P-FET devices, and/or combinations thereof as known in the art. In suchan embodiment, charge evacuation element 3380 includes a chargeevacuation gate 3381, a charge evacuation drain 3383, and a chargeevacuation source 3382.

As used herein, a reset element 3350 can be any device capable ofswitching, such that voltage source 3345 is applied to light sensitiveelement 3390. In one particular embodiment, reset element 3350 is a CMOStransistor that includes a reset gate 3351, a reset drain 3353, and areset source 3352. Yet further, for purposes of this document, lightsensitive element 3390 is any device that is capable of detecting thepresence of light and producing a signal indicative of the amount oflight detected. Thus, in one embodiment of the present invention, lightsensitive element 3390 is a photodiode. In other embodiments, lightsensitive element 3390 is a photo-gate. Other such light sensitiveelements can be used including, but not limited to, a bipolarphoto-transistor and/or a stacked complex n-p-n-p device where eachjunction is sensitive to a part of the light spectrum to be detected.

Also for purposes of this document, selection element 3370 can be anydevice capable of presenting a signal representative of an amount oflight impinging upon light sensitive element 3390. Thus, selectionelement 3370 can be a transistor, a pass gate, or the like. In oneembodiment of the invention, selection element 3370 is a CMOS transistorwith a selection gate 3371, a selection drain 3373, and a selectionsource 3372.

For discussion purposes, each of reset element 3350, source followertransistor 3360, selection element 3370, and charge evacuation element3380 are CMOS transistors and light sensitive element 3390 is aphotodiode. However, based on the disclosure provided herein, one ofordinary skill in the art will recognize that the present invention andthe principles included herewith are applicable to a number of otherdevice types.

As illustrated by pixel 3000, source follower drain 3363 and reset drain3353 are both electrically coupled to voltage source 3345. Further,source follower gate 3361, a node of light sensitive element 3390, resetsource 3352, charge evacuation source 3382, and charge evacuation drain3383 are electrically coupled together at a node 3303. The other node oflight sensitive element 3390 is electrically coupled to common ground3355. Yet further, source follower source 3362 is electrically coupledto selection drain 3373 at a node 3313.

Reset gate 3351 is driven by im_line_rst(i) 707, charge evacuation gate3381 is driven by im_line_rst_n(i) 717, and selection gate 3371 isdriven by im_line_sel(i) 727. Output signal 742 is driven by selectionsource 3372. In some embodiments, im_line_rst_n(i) 717 is the inverse,or complement, of im_line_rst(i) 707. Thus, when reset element 3350 isswitched, thereby providing a low impedance path from reset drain 3353to reset source 3352, charge evacuation element 3380 is not switched,thereby providing an open circuit between charge evacuation drain 3383and charge evacuation source 3382. The opposite is also true. When resetelement 3350 is not switched, thereby providing an open circuit betweenreset drain 3353 and reset source 3352, charge evacuation element 3380is switched thereby providing a low impedance path from chargeevacuation drain 3383 to charge evacuation source 3382.

In some embodiments where im_line_rst_n(i) 717 is the complement ofim_line_rst(i) 707, im_line_rst_n(i) 717 is delayed such that a fallingedge of im_line_rst(i) 707 precedes a corresponding rising edge ofim_line_rst_n(i) 717 by a period of time. In other embodiments,im_line_rst_n(i) 717 works in relation to im_line_rst(i) 707, but is notthe complement of im_line_rst(i) 707, but rather a distinctly generatedsignal. The timing relationships of the various signals are discussedfurther in relation to FIGS. 9 below.

FIG. 8 illustrates a flow diagram 3400 of one method in accordance withthe present invention for operating pixel 3000. In operation,im_line_rst(i) 707 is asserted such that a low impedance path existsbetween reset drain 3353 and reset source 3352 (block 3410). Inparticular embodiments, im_line_rst(i) 707 overdrives reset gate 3351such that photodiode 3390 is charged as rapidly as possible. Further,im_line_rst_n(i) 717 is deasserted such that an open circuit existsbetween charge evacuation drain 3383 and charge evacuation source 3382(block 3420). As such, a voltage corresponding to voltage source 3345,less the impedance drop across reset element 3350 is present at node3303, thereby reverse biasing photodiode 3390. In this state, photodiode3390 is charged to a level corresponding to the voltage at node 3303.Further, a negative charge is built in the channel of reset element3350.

Some time later, im_line_rst(i) 707 is deasserted such that an opencircuit exists between reset drain 3353 and reset source 3352 (block3430). Thus, additional charge from voltage source 3345 is not availableto photodiode 3390. However, on the falling edge of im_line_rst(i) 707,the negative charge accumulated in the channel of reset element 3350 isdissipated through both reset drain 3353 and reset source 3352. Thus,part of the charge is discharged via node 3303 causing a voltage drop(i.e., noise), which can be significant. In some cases, the voltage dropdue to the negative charge from reset element 3350 can be betweentwo-hundred and five-hundred millivolts for geometries of four to sevenmicrometers, and is exacerbated as pixel geometries and source voltagesdecrease.

With an open circuit between reset drain 3353 and reset source 3352, thecharge built up in photodiode 3390 begins to dissipate at a ratecorresponding to the amount of light impinging upon photodiode 3390.Thus, where a significant amount of light impinges upon photodiode 3390,the voltage at node 3303 will decrease at a more rapid rate than if asmall amount of light impinges upon photodiode 3390.

Either some time after the deassertion of im_line_rst(i) 707 orcoincident therewith, im_line_rst_n(i) 717 is asserted such that a lowimpedance path exists between charge evacuation drain 3383 and chargeevacuation source 3352 (block 3440). This assertion provides a pathwhereby at least a portion of the negative charge that was stored in thechannel of reset element 3350 can dissipate through charge evacuationdevice 3380. In some embodiments, the type of device used to implementcharge evacuation element 3380 is chosen such that the operation ofcharge evacuation element 3380 reflects the non-linearities exhibited byreset element 3350. Thus, in some cases, charge evacuation element 3350and reset element 3350 are chosen to be the same device types.

As approximately half of the charge built up in the channel of resetelement 3350 dissipates through reset drain 3353 and the other halfdissipates through reset source 3352, the size of charge evacuationdevice 3380 can be chosen such that it can evacuate roughly half of thecharge built up in the channel of reset element 3350. Thus, in oneembodiment, where the charge evacuation element 3380 and reset element3350 are the same device type, the size of charge evacuation device 3380is roughly half the size of reset element 3350. In other embodimentswhere reset element 3350 and charge evacuation element 3380 aredifferent device types, such as, when charge evacuation element 3380 isa switched capacitor, the size of the capacitor is chosen based on theamount of charge to be dissipated from the channel of reset element 3350via reset source 3352. As illustrated below in FIGS. 9, dissipation ofthe negative charge in the channel of reset element 3350 through chargeevacuation element 3380 can, depending upon the geometry of pixel 3000,eliminate as much as five-hundred millivolts of noise at node 3303.Elimination of such noise at node 3303 assures that the signal sampledat output 742 more accurately reflects the amount of light detected atphotodiode 3390.

At some point, im_line_sel(i) 727 is asserted to sample the remainingvoltage at node 3303, and thereby gain an approximation of the lightimpinging upon photodiode 3390 (block 3450). With im_line_sel(i) 727asserted, a low impedance path exists from selection drain 3373 toselection source 3372. Thus, a signal representative of the voltagepresent at node 3303 is presented on output 742 via source followertransistor 3360. Once the output is sampled, im_line_sel(i) 727 isdeasserted (block 3460) and the process is repeated to detect the amountof light impinging upon photodiode 3390 at a future point in time. Itshould be recognized from FIGS. 9 that in some embodiments of thepresent invention, the assertion of im_line_sel(i) 727 overlaps theassertion of im_line_rst(i) 707. Yet, in other embodiments, theassertion of im_line_sel(i) 727 does not overlap the assertion ofim_line_rst(i) 707.

FIGS. 9 illustrate timing diagrams of the various signals related to theoperation of pixel 3000. More particularly, the figures illustrate therelationship between im_line_rst(i) 707, im_line_rst_n(i) 717,im_line_sel(i) 727, the voltage at node 3303, and the voltage at output742.

FIG. 9 a is a timing diagram 3501 illustrating operation of pixel 3000where im_line_rst(i) 707 and im_line_rst_n(i) 717 are complementary, andwhere charge evacuation element 3380 is operational. In contrast, FIG. 9b is a timing diagram 3502 illustrating similar operation of pixel 3000where charge evacuation element 3380 is not present. As illustrated inFIG. 9 b, at the falling edge of im_line_rst(i) 707 (noted as 3520), avoltage drop, VDROP 3510, is exhibited at node 3303. This voltage dropexists until photodiode 3390 is again charged by assertion ofim_line_rst(i) 707 (noted as 3530). This voltage reduction at node 3303(i.e., noise) causes a corresponding reduction in output signal 742.This is in comparison to timing diagram 3501 of FIG. 9 a, where thevoltage drop at node 3303 does not occur because the charge causing thevoltage drop in timing diagram 3502 is evacuated through chargeevacuation element 3380 as previously described. Thus, as illustrated inthe contrast between FIG. 9 a and FIG. 9 b, a substantial amount ofnoise can be eliminated from node 3303 via the operation of chargeevacuation element 3380.

FIG. 9 c is a timing diagram 3503 that illustrates another embodimentwhere, similar to timing diagram 3501, im_line_rst(i) 707 andim_line_rst_n(i) 717 are complementary. However, in contrast to timingdiagram 3501, timing diagram 3530 illustrates im_line_rst_n(i) 717delayed from im_line_rst(i) 707 by a period 3570. As illustrated, chargeevacuation element 3380 still dissipates the charge from the channel ofreset element 3350, albeit at a slightly later time corresponding to theassertion of im_line_rst_n(i) 717. Thus, a voltage droop 3580corresponding to VDROP 3510 (illustrated as a dotted line) is noted onnode 3303, until the assertion of im_line_rst_n(i) 717 (noted as 3590).However, because of the operation of charge evacuation element 3380, thevoltage at node 3303 is corrected before the sample is taken uponassertion of im_line_sel(i) 727.

Based on the disclosure provided herein, one of ordinary skill in theart will recognize a number of other timing relationships between thevarious signals that each dissipate the charge within the channel ofreset element 3350, thus reducing the noise manifest at output 742.Further, as previously mentioned, im_line_rst(i) 707 andim_line_rst_n(i) 717 can be separately generated to achieve a timingrelationship that allows for charging of photodiode 3390, and fordissipating the charge within the channel of reset element 3350. In somecases, it can minimize the logic in TCAG 136 to have im_line_rst_n(i)717 be the complement of im_line_rst(i) 707. Further, it should berecognized that im_line_sel(i) 727 can be generated at other times thanthose illustrated in FIGS. 9.

FIG. 10 illustrates the architecture of column amplifier 703. Discharge(element 777) and enable (element 767) signals are received from columnlogic block 739. Using column amplifier 703, the output of each columnis double sampled by the track and hold circuits in the column amplifierthat are controlled by im_smp_strobe (element 747) and im_ref_strobe(element 757) signals. A timing diagram 893 illustrating theinter-relationship of the various signals of pixel array 700 and columnamplifier 703 is provided as FIG. 11. Of note, the exposure controldescribed in more detail below should not have a line in pixel array 700during a period 892 outlined by dotted lines on FIG. 11.

Imaging unit 130 can be programmed to provide a standard output frompixel array 700 or a mirrored image. FIG. 12 provides a timing diagramof the signals associated with outputting the image in standard mode,and FIG. 13 provides a timing diagram of the signals associated withoutputting the image in mirrored mode. Selection between the standardand mirrored modes is accomplished by programming an im_mirror (element787) signal via a register bit, which drives column logic block 739(FIG. 4).

Referring again to FIG. 4, analog mux 709 selects the pixel from pixelarray 700 that will be processed. Thus, a column of pixel array 700 isselected by column logic block 739, and then a single pixel within thecolumn is selected via analog mux 709. Analog mux 709 is a three levelmux where the first and second levels are eight to one muxes, and thethird level is a six to one mux. This provides for a total of a 384 toone mux. Selection is accomplished via column address signals (element797) im_col_add_L[7:0] for selecting the first level mux,im_col_add_M[7:0] for selecting the second level mux, im_col_add_H[5:0]for selecting the third level mux.

Sensor 132 can operate in a rolling mode, in which the exposure andreadout processes are linked. The sampling of each row data is performedat the end of the integration time. After sampling a selected-row data,the row is reset. The row is sampled again after the reset. This secondsampling helps reduce fixed pattern noise (FPN) by implementing adouble-sampling procedure. The integration of that row starts againaccording to the exposure parameters.

A set of registers defines the total size of pixel array 700 and thesize of the array that is read out and sent to translation unit 150 forfurther processing. Another set of registers defines the exposure timethat is required for pixels within pixel array 700. The exposure andread timing sequences are derived from these registers.

The processing of a single pixel of sensor 132 is referred to as a pixelcycle. There are four Rclk 601 cycles in a pixel-cycle that iscontrolled by a timing circuit. An Im_Pix signal from clock unit 102qualifies the first clock edge of the pixel-cycle. On activation of thetiming circuit, line and pixel counters are set to zero, andsubsequently begin counting pixel-cycles. An Im_Frame 910 signal isactivated on the first pixel-cycle of the frame, and an Im_Line 920signal is activated on the first pixel-cycle of a line, except for thefirst line, when Im_Frame 910 is active.

The pixel counter counts upon activation of the Im_Pix signal, and theline counter is activated when the pixel counter reaches a full-linecount. On activation of Im_Line 920 after activation of Im_Frame 910,TCAG 136 reads and resets the first line in pixel array 700. On eachactivation of Im_Line 920, the lines are only reset.

Once the line whose index is equal to the Im_starty register value isreached, data is sent out starting at the pixel whose index matches theIm_startx register value. The pixels that are inside the window definedby the Im_starty and Im_startx, Im_endy, and Im_endx converted from theanalog to digital domain using ADC 134, and passed to translation unit150.

An additional window can be defined for differentiating between pixelsin “area of interest”, and pixels outside this defined area. AnIy_inwindow signal is active for those pixels inside the window, andinactive for the pixels outside the window.

Im_Line 920 starts the readout process of an image line, and also causesa reset to that line. One or more register values define the reset timefor the first line. Further, a coarse exposure register value definesthe number of lines between Im_Frame 910 signal activation and the endof the coarse reset time. Once the appropriate Im_Line 920 is reached,the number of pixel-cycles is counted according to a fine exposureregister value. Once the reset time has elapsed, the integration time ofthe first line is initiated. The event is marked by the activation ofIm_Expline 930 for one pixel-cycle. A second nine-bit counter is thenloaded with the full-line length register and decremented on each pixelcycle. Once the nine-bit counter reaches a zero value, the integrationtime of the second line is initiated. Im_Expline 930 is activated forone pixel-cycle and the process repeats itself until the integrationtime of the last line is initiated. FIG. 14 illustrates the interactionof the previously described frame timing signals.

Each activation of Im_Expline 930 starts the exposure of an image line,and Im_Line 920 signal starts the readout and reset processes. Asillustrated in FIG. 15, the line is sampled twice, once at the end ofthe exposure time (element 1010), and for the second time right afterthe activation of the reset signal (element 1020).

Double buffering can be used for all frame parameters. This allows theHost or programmable processing core 112 to write the next frameparameters at any time. Implementation of the newly written parametersare, however, delayed until activation of the following Im_Frame 910 toavoid problems with the currently processing frame.

Referring again to FIG. 4, a signal created by a selected pixel withinpixel array 700 is passed through analog mux 703 and source selectionmux 709 prior to further processing by sensor 132. More particularly,the selected pixel signal is level shifted by level shift block 719,black level corrected by black offset block 723, and amplified by firststage programmable gain amplifier 729, and second stage programmablegain amplifier 733. The output of the amplified and corrected pixelsignal is then passed from sensor 132 to ADC 134.

In one particular embodiment of imaging device 100, the level shift, andblack offset are done in the digital domain. In other embodiments, theprocesses are performed in the analog domain, while in yet otherembodiments, the level shift and amplification are performed in theanalog domain, while the black offset is performed in the digitaldomain. Thus, it should be recognized that the point at which a selectedpixel signal is presented to ADC 134 can be changed depending upon theparticular embodiment. Further, the order in which amplification, blackoffset and level shifting are performed can be modified. For example,the selected pixel signal can be level shifted and amplified in theanalog domain, passed to ADC 134, and subsequently black level correctedby black offset block 723 in the digital domain.

As further discussed below, analog signal processing portions of imagingdevice 100 are designed to ensure that as much as possible of an imagedetected by pixel array 700 is passed from the analog domain to thedigital domain. Further, power and bandwidth considerations areentertained in deciding whether the processing is performed in theanalog or digital domain. Thus, the order of processing through ADC 134,level shift block 719, black offset block 723, amplifiers 729 and 733,as well as whether the blocks are implemented in the analog or digitaldomain can be adjusted to ensure that the entire resolution of ADC 134is exploited, adjusted based upon power and bandwidth considerations,and/or a combination thereof.

As used herein, dark level is the level of the pixel signal due toleakage and other circuit related effects that occurs in unexposed pixelduring a given exposure time. The dark level can be obtained by readingthe pixels of the dark rows. Black level is the level of the pixelsignal corresponding to the darkest pixel in a frame (e.g., shadowedareas of an image). White level is the level of the pixel signalcorresponding to the brightest pixel in a frame (e.g., highlighted areasof an image).

As illustrated in FIG. 4, the selected pixel signal passed to ADC 134 isdescribed by the equation:

$V_{adc} = {\left\{ {{\left( {V_{mux} - {\Delta\; V_{ref}}} \right)*\left( {1 + {{im\_ lsh}{\_ gain}}} \right)} - {\left( {- 1} \right)^{{im\_ black}{\lbrack 4\rbrack}}\frac{\Delta\; V_{ref}{{im\_ black}\left\lbrack {3\text{:}0} \right\rbrack}}{16}}} \right\}*\frac{2\left( {1 + {{im\_ gain}{{\_ m}\left\lbrack {2\text{:}0} \right\rbrack}}} \right)}{1 + {{im\_ gain}{{\_ d}\left\lbrack {1\text{:}0} \right\rbrack}}}}$

The signals in the analog processing chain are differential voltages inthe range negative one volt to positive one volt. The uncorrected andun-amplified output from source selector mux 713 is in the range of zerovolts to one and one-half volts. Level shift block 719 converts theoutput from source selector mux 713 to a voltage range compatible withthe other analog processing circuits. In some embodiments, level shiftblock 719 is operable in one of two different modes: a high gain modeand a high dynamic range mode. The high gain mode is selected by settinga register output im_lsh_gain 714 equal to a high level. The high gainmode operates by mapping a zero to one volt input range to a negativeone volt to positive one volt range.

Alternatively, the high dynamic range mode is selected by setting aregister output im_lsh_gain 714 equal to a low level. The high dynamicrange mode operates by mapping a zero to two volt input range to thenegative one volt to positive one volt range. FIG. 16 illustrates thecorrection and amplification of the selected pixel signal as it isreceived from source select mux 713 and passed through level shift block719, black offset block 723, and amplifier stages 729, 733 for both highgain mode (signal 844) and high dynamic range mode (signal 846).

For a given exposure time, the voltages of most pixels in the frame willbe in the range between the black level and the white level. In someembodiments, ADC 134 ideally converts only this voltage range, withoutlosing resolution by eliminating signals outside of the range. In oneparticular embodiment, adjusting the voltages of the pixels to be in therange between black level and white level is achieved by black offsetblock 723. To this end, a control signal, im_black[4:0] (element 724),can be provided in sign-magnitude format and use to program the leveladjustment to the proper level. In one embodiment, the resolution ofblack offset block 723 is equivalent to the thirty-two least significantbits of ADC 134.

FIG. 17 illustrates examples of signal outputs for normally lighted(signal 853), dark (signal 857), and bright (signal 859) subjects fromsource select mux 713 that are passed through level shift block 719,black offset block 723, and amplifier stages 729, 733. The gain andblack level values are all pixel color dependent. The red-green andblue-green ratios of the gain settings are determined in order toachieve gross white balance.

An exemplary timing of the control signals for the analog processingcircuits is presented in FIG. 18. A sel[1:0] (element 774) isrepresentative of the timing of column address signals 797 and also andim_test[7:0] (element 704) signal. In one particular embodiment, amaximum exposure time is one frame at thirty frames per second, or 33.33milliseconds, and a maximum gain of the analog chain is thirty-two (i.e,two from level shift block 719, and four from each of amplifiers 729 and733). A pixel sensitivity of 0.226 [V/lux*sec] for a green pixel at 540nm is provided by pixels within pixel array 700. Full scale of ADC 134is two volts (e.g., negative one volt to positive one voltdifferential), where the least significant bit is 1.95 mV.

Using these parameters, the size of the analog processing section ofimaging device 100 can be determined and optimized. For determining theresolution used to obtain the proper exposure, the lowest illuminationlevel that can be translated to full scale response of ADC 134corresponds to the maximal exposure time and maximal amplifier gain. Thefollowing equation describes the approach:2[V]/(32*0.0333[sec]*0.226[V/lux*sec])=8.3[lux]

If the highest level of illumination is 100,000 [lux], the lowestexposure time needed assuming minimal amplifier gain of unity is:2[V]/(1*100,000[lux]*0.226[V/lux*sec])=88.5[usec]

As in a CIF frame there are 101,376 pixels, the minimal exposure timecorresponds to about three-hundred, fifty-four pixel cycles at thirtyframes per second. Therefore, controlling exposure time at theresolution of one pixel cycle is adequate.

For sizing black offset block 723, it is assumed that the exposure isset so that at the output source select mux 713, the white levelcorresponds to a drop of one volt and the black level corresponds to adrop of 937.5 mV. If the amplifier 729, 733 gain is set to sixteen, andthe black level DAC is five-bit (sign magnitude) with the leastsignificant bit equaling 62.5 mV is set to fifteen, this will stretchthe narrow histogram on the full resolution of ADC 134.

On the opposite end, for a dark picture where at the output of sourceselect mux 713 the black level is zero and the white level correspondsto a drop of 62.5 mV, a setting of negative fifteen for the black levelDAC and a gain of sixteen for the amplifier 729, 733 will stretch thenarrow histogram on the full resolution of ADC 134.

The programmable gain amplifier 729, 733 has a maximum gain of sixteenand supports several functions including extending the exposure rangefor low illumination levels, providing gross white balance, andstretching the histogram as described in relation to black levelcorrection. Programmable gain amplifier 729, 733 can be implemented as atwo stage amplifier, where each stage exhibits a maximum gain of four.First stage programmable gain amplifier 729 is described by thefollowing equation:

${Gain}_{PGA\_ D} = \frac{4}{1 + {{im\_ gain}{{\_ d}\left\lbrack {1\text{:}0} \right\rbrack}}}$

Second stage programmable gain amplifier 733 is described by thefollowing equation:

${Gain}_{PGA\_ M} = \frac{1 + {{im\_ gain}{{\_ m}\left\lbrack {2\text{:}0} \right\rbrack}}}{2}$

There is separate gain control for each color filtered pixel. Theparameters associated with the gain control are used for coarse whitebalance, and for enhanced exposure in dim light situations, where themaximum exposure time has been reached, but the output image is stilltoo dark. The gain parameters are used for expanding the range of thesampled data to the full range of ADC 134.

The first sixteen lines of pixel array 700 are dark lines that are notexposed to light. Each of the lines are integrated similar to otherlines and the dark level for each color is separately measured on theoutput of ADC 134. The average of the measured dark line values ispassed back for Dark pixel correction of the lines that follow the darklines in pixel array 700. The effect can be immediate, where thecalculation result is activated during the readout of the same framewhere the dark lines were evaluated.

Image device 100 includes two methods for calculating the dark offset.One or the other mode is selected by writing a predetermined modeselection in a register. The two modes are dark offset per column andaverage dark offset. Dark offset per column involves calculating andstoring an offset value for each column. When reading the image area,the dark offset parameter selected based on the particular column thatis currently being read from pixel array 700, where the combination ofline number and column number indicates the color being read.

The offset for each column is averaged on the fifth to twelfth lines,giving a total of eight lines. The offset for each color component ineach column is calculated as the average of four pixels of the samecolumn and color, minus the fixed black offset that is applied duringthese lines. During the dark offset calculation, the black offset inputsare set to a positive known value and stored in a register, the colordependent gain is applied, and the output of ADC 134 is sampled.

Two RAMs of 384 by twelve-bits can be used. The output from ADC 134 forthe first and second dark lines is written to the RAMs. For thefollowing lines, the even-numbered line values are added to the firstRAM entries, and the odd-numbered line values to the second RAM entries.After the seventh line (a “RED” line), values are added and the resultis stored back in the RAM. The next line is processed in the same mannerfor the “BLUE” line offset. Throughout this process, the average offsetper color is calculated as well.

During the period for lines twelve and thirteen, which are dark linesthat are not read from the array, the RAM contents are read, and the“column noise” is calculated as follows:Column_offset=(RAMsum)/4, rounded upNoise=Column_offset−Average, 12 bits two's complement.

When reading the image pixels from pixel array 700, the “noise” darkoffset is read from the RAM. The correction is performed in the digitaldomain on the output of ADC 134. The values that are stored in the RAMare subtracted from the output of ADC 134 according to the column numberand the line type that is being read.

To simplify calculations, average dark offset mode is utilized. In theaverage dark offset mode, the first four lines are skipped, and only thenext eight dark lines are evaluated. Evaluation is performed from pixelnumber sixty-four to pixel number three hundred nineteen, totaling 256pixels per line. A total of 128×2 green, 128 blue and 128 red pixelvalues are collected. The result is averaged, limited, and stored inregisters.

Further, in some embodiments, three four-bit black offset parameters arestored in a register for each of the three colors. The value in theregisters are passed to imaging unit 130 and subtracted from or added tothe analog image signal before the signal is passed to ADC 134. Thus,three four-bit digital to analog converters (“DAC”) are used to convertthe register values to the analog domain before the subtraction processis accomplished. The operation is performed in accordance with a signbit that is passed along with each of the three register values. In someinstances, the four-bit offset values are each multiplied by sixteenbefore the subtraction process is completed.

In addition, some embodiments of imaging device 100 include provisionsfor reducing random noise. Such provisions include, but are not limitedto, applying the aforementioned charge injection technique to pixelreset signals, stopping operation of the various digital circuitry inimaging device 100 while each row of pixel array 700 is sampled, andproviding a clock driving the analog circuitry that precedes the clockdriving the digital circuitry to keep various substrate noise effectsorthogonal in time domain. In one embodiment, such a phase differencebetween the analog clock and the digital clock is provided by derivingthe analog clock from the falling edge of the digital clock, andsubsequently adding some delay to the analog clock before applying it toADC 134.

Some embodiments include additional measures to eliminate random noisethat include, but are not limited to, providing analog processingswitched capacitor circuits that use clock phases generated by ADC 134and synchronizing control signals driving the analog circuitry using theclock generated by ADC 134. In addition, the sizes of the capacitorsused in the switched capacitor circuits can be increased in order tomeet the precision requirements and to reduced kT/Cnoise.

Some embodiments of the present invention provide for reducing powerconsumption of imaging device 100. Such processes can include reducingpower consumption of the analog processing circuitry of imaging device100, while in other embodiments, power consumption is reduced byreducing power consumption of the digital processing circuitry ofimaging device 100. In one particular embodiment, power consumption isreduced by limiting power consumption by both the analog processingcircuitry and the digital processing circuitry of imaging device 100.

FIG. 19 is a timing diagram illustrating the relationship between powerconsumption control signals, Im_stdby 764 and im_adc_cnt[4] 754. Aspreviously illustrated in FIG. 4, Im_stdby 764 is provided to thevarious analog processing circuits of sensor 132 including analog mux709, source selection mux 713, level shift block 719, black offset block723, and programmable gain amplifier 729, 733. Im_adc_cnt[4] 754 isprovided to ADC 134. Following FIG. 19, when Im_stdby 764 is assertedhigh (noted 1920), the analog processing circuitry stops functioningduring a non-functional period 1910 surrounded by dashed lines. Onecycle of aclk 603 after assertion of Im_stdby 764, Im_adc_cnt[4] 754 isasserted high (noted 1940), thus placing ADC 134 in the standby mode. Toawake from the standby mode, Im_adc_cnt[4] 754 is deasserted (noted1950) and twenty cycles of aclk 603 later (noted 1960), Im_stdby 764 isdeasserted (noted 1970). Once Im_stdby 764 is deasserted, the analogprocessing circuitry becomes operational again one cycle of aclk 603later (noted 1980).

During non-functional period 1910, the analog processing circuits willenter a low power consumption standby mode. In one embodiment of thestandby mode, the common mode feedback of the operational amplifiersremains operational, but the circuits associated with the various analogprocessing calculations are disabled. This maintains the inertial stateof the amplifiers, while reducing the power consumption of the analogcircuitry associated with imaging device 100.

FIG. 20 illustrates a block diagram 1905 of circuitry related tocontrolling the standby mode. Block diagram 1905 includes pixel array700 providing differential inputs 1903 to a processing control system1925. Processing control system 1925 provides to an analog processingcircuit 1955 via differential inputs 1906, either differential inputs1903 or a differential reference voltage 1915 (received via inputs1904). Differential inputs 1906 are processed by analog processingcircuitry 1955 as previously discussed, and the results presented viadifferential inputs 1907 to ADC 134 are converted to the digital domainand passed to a digital processing circuit 1975 via a bus 1965. Digitalprocessing circuit 1975 performs the various digital processingfunctions as previously discussed, and the processed data is provided asan image output.

In some embodiments, analog processing circuit 1955 includes columnamplifier 703, analog mux 709, source selection mux 713, level shiftblock 719, black offset block 723, and programmable gain amplifier 729,733. In other embodiments, analog processing circuit 1955 includes asubset of the aforementioned element. For example, in one particularembodiment, analog processing circuit 1955 includes level shift block719, black offset block 723, and programmable gain amplifier 729, 733.Thus, depending upon the design, one or more of the aforementionedblocks can be placed in standby mode either individually, or as a group.Further, in some embodiments, digital processing circuit 1955 includestranslation unit 150 and parallel interface unit 190. In otherembodiments, digital processing circuit 1955 includes only one or theother of translation unit 150 and parallel interface unit 190.

In some cases, only a subset of the pixels within pixel array 700 areprocessed and provided as an image output. For example, bandwidthavailable for the image output may be less than that required to outputeach pixel of pixel array 700. As a more specific example, every otherpixel line and/or pixel column of pixel array 700 may be dropped. Insuch cases, embodiments of the present invention provide for limitingpower consumption of imaging device 100 by placing analog processingcircuit and/or digital processing circuit 1975 in the standby mode. Todo this, a sub-sample control system 1945 provides an indication aboutwhether a particular line, column and/or pixel is to be processed. Thisindication from sub-sample control system 1945 is used to generateIm_stdby signal 764 which, as previously discussed, controls whetheranalog processing circuit 1925 and/or digital processing circuit 1975are to enter the standby mode, or remain operational.

In some embodiments, sub-sample control system 1945 is a hardware systemutilizing information from frame rate block 114 of control unit 110and/or TCAG 136 of imaging unit 130 to determine where in a frame areceived pixel is from, and whether the pixel is to be processed.Alternatively, in other embodiments, sub-sample control system 1945 isimplemented in software and executed by programmable processing core112. In yet another embodiment, sub-sample control system 1945 is ahybrid hardware and software implementation utilizing relevant portionsof TCAG 136, frame rate block 114, and/or programmable processing core112.

In one particular example where only two of every four pixel lines areprocessed, sub-sample control system 1945 monitors the addressing ofpixel array 700 and causes Im_stdby 764 to assert each time anon-processed pixel line is addressed. This causes processing controlsystem 1925 to present reference voltage 1915 to analog processingcircuit 1955 in place of differential input 1903 from pixel array 700.Reference voltage 1915 is electrically coupled to switching inputs ofanalog processing circuit 1955, thus reducing power dissipation due toswitching circuits in analog processing circuit 1955. However, thecommon mode feedback of the operational amplifiers within analogprocessing circuit 1955 continue operating. This maintains the inertialstate of the amplifiers, while reducing the power consumption of theanalog circuitry associated with imaging device 100.

In addition, Im_adc_cnt[4] 754 is asserted to place ADC 134 in a similarstandby mode. Yet further, a digital disable signal 1985 is providedfrom processing control system 1925 to digital processing circuit 1975.This signal is used to gate the various clocks distributed to thevarious operating units of imaging device 100. In some embodiments,processing control system 1925 comprises portions of Rclk unit 102. Insuch embodiments, the various clock signals are gated by Rclk unit 102and there is no reason to provide digital disable signal 1985 to digitalprocessing circuit 1975.

FIG. 21 a illustrates an exemplary circuit 1943 used to electricallycouple analog circuits within analog processing circuit 1955 toreference voltage 1904. In circuit 1943, PH1 and PH2 are non-overlappingclock phases. In standby mode, there is no switching to vref 1904 a andvref_n 1904 b, rather inputs 1903 are shorted to vcommon 1953 to limitswitching. With such a circuit, both the bias currents of the Op-Amps,as well as the switching related to analog calculations is reduced.However, switching sufficient to keep the common mode feedback loops ofan amplifier 1963 active is maintained. When Im_stdby 764 is assertedhigh, standby switches 1964 are shut and operational switches 1965 areopen. Thus, switching in circuit 1943 is limited.

FIG. 21 b illustrates exemplary circuit 1943 when Im_stdby 764 isdeasserted, and circuit 1943 is operating in phase one, PH1. FIG. 21 cillustrates exemplary circuit 1943 when Im_stdby 764 is similarlydeasserted, but circuit 1943 is operating in phase 2, PH2.

FIG. 22 is a flow diagram 1974 illustrating a method according to thepresent invention for entering and exiting the standby mode. Followingflow diagram 1974, parameters associated with sub-sample control system1945 are set-up (block 1976). This can include writing various registerscontrolling the portions of pixel array 700 that are processed. Such aregister write can be performed by programmable processor control 112and/or the Host. An image is detected from pixel array 700 (block 1977).Pixel array 700 is addressed by TCAG 136 to cause a desired column ofpixel signals to be presented by pixel array 700 (block 1978). Ifsub-sample control system 1945 indicates that the column is to beprocessed (block 1979) and that the line is to be processed (block1981), then the analog processing circuitry 1955 is enabled, or awakenedfrom the standby mode (block 1982). Further, digital processingcircuitry 1975 is enabled by removing the clock gate (block 1983). Withthe circuitry thus enabled, the pixel signals received from pixel array700 are processed.

Alternatively, where either the pixel column (block 1979) or the pixelline (block 1981) is not to be processed, analog processing circuit 1955(block 1986) and digital processing circuit (block 1987) are placed inthe standby mode to conserve power. In such a state, the pixel signalsreceived from pixel array 700 are dropped and not processed (block1988).

The power distribution and management scheme for the analog circuitryprovides a supply of 2.8 volts+/−10%. On-chip regulators are used toprovide the following supply voltages: _(—)2.5 volts+/−10% for the 0.25u digital circuits, 2.4 volts+10/−5% for the imager array on a vddArr881 supply, and 3.2 volts+10/−5% for the imager array line resetdrivers. A block diagram 896 of the power management for imaging device100 is illustrated in FIG. 23. As illustrated, the regulators arecontrolled by the following signals: a reg_d_enable 882 signal the whenasserted low causes the digital regulator to enter standby mode with the2.5 V voltage is supplied from an external source. In order to enablepower up, reg_d_enable 882 is gated with an atest 883 signal as shown inFIG. 23. At power up atest 883 is kept low forcing reg_d_enable 882high. When reg_d_enable is 882 is high, the digital regulator outputvoltage is reduced to 2.0 V. The order of power-up of the regulators hasto be digital regulator 884 first and only then a boost regulator 885.The clock of boost regulator 885 has to be provided by the digitalcircuits. It is a clock in the range of 500 KHz to 1 MHz, obtained bydividing Rclk 601 by sixteen. The edges of the divided clock should bealigned with the digital signals (i.e., aligned with the rising edge ofRclk 601).

To facilitate testing and/or characterization, imaging device 100includes one or more built in self test (“BIST”) routines. In particularembodiments, the BIST routines provide the ability to test one or moreof blocks 700, 703, 709, 713, 719, 723, 729, 733, and/or 134 using adigital tester. In some cases, such testing can further be accomplishedwith or without illumination of imaging device 100. Such approaches canreduce the costs and increase the rate at which the functionality ofimaging device 100 is verified.

In some embodiments, one or more additional lines of “pseudo”-pixels areincluded within pixel array 700. The output from the pseudo-pixels canbe a pre-defined voltage that remains constant regardless of any lightimpinging upon pixel array 700. For example, in one instance, thepseudo-pixels have an output of one volt that is passed to the columnamplifiers. These pixels are activated by asserting im_line_test(element 737). The outputs of the pseudo-pixels are passed throughcolumn amplifier 703, and analog mux 709. In turn, the outputs arepassed though source selector mux 713 by proper setting of im_test[8:0](element 704). Im_test[8:0] (element 704) is derived from decoding aprogrammable digital register (im_test_en[3:0]). The outputs are thenpassed through one or more of blocks 719, 723, 729, 733, and 134. Theoutput of ADC 134 can then be compared against a range of values todetermine if imaging device 100 is within acceptable design limits. Thecomparison can be performed either by a programmable hardwarecomparator, a hardwired hardware comparator, or via programmableprocessor core 112. Such a test provides an ability to test theoperation of column amplifier 703, and/or one or more of blocks 719,723, 729, 733, and 134. As the inputs and outputs associated with thetest are available in the digital domain, a purely digital test of theanalog circuit can be performed. This provides an efficient and costeffective means for verifying the functionality of imaging device 100 astesting of the analog circuitry can be performed on a digital tester.

As previously stated, an output to be tested using one or more of theBIST routines can be passed through one or more of blocks 719, 723, 729,733, and 134. Which blocks the outputs pass through is programmable.Using such a programmable approach, each block can be individuallybypassed. Such bypassing allows for the output of each blockindividually, or a combination of the blocks to be tested to be isolatedfor testing purposes. Bypassing of the individual blocks is controlledby signals im_by_lsh 702 to bypass level shift block 719, im_by_blk 712to bypass black offset block 723, im_by_pgd 722 to bypass first stageprogrammable gain amplifier 729, and im_by_pgm 732 to bypass secondstage programmable gain amplifier 733.

Further, in some embodiments, source selector mux 713 permitsapplication of various test voltages to the input of the analogprocessing circuitry. This provides for applying known analog voltagesto one or more portions of analog processing blocks 719, 723, 729, 733,134. In one embodiment, the digital output generated by applying theknown analog voltage to the blocks is compared against a predeterminedvalue, or range of values. In this way, testing of analog blocks 719,723, 729, 733, 134 can be performed as a purely digital test, whereim_test (8:0) (element 704) is generated from a programmable digitalregister (im_test_en[3:0]), and the outputs are each tested in thedigital domain. Again, this provides an efficient and cost effectivemeans for verifying the functionality of imaging device 100 as analogtesting can be performed on a digital tester. Further, as each of blocks719, 723, 729, 733 can be individually bypassed, any combination ofblocks 719, 723, 729, 733 can be verified using the digital test.

In one particular embodiment of the present invention, source selectormux 713 selects between five input voltages based on im_test(8:0)(element 704). More particularly, source selector mux 713 permitsconnecting reference voltages (v_ref 706, common 716, and vref_n 726)that are generated internal to imaging device 100 to the input of theanalog processing blocks. In some embodiments vref 706 and vref_n 726are derived from ADC 134, and common 716 is a provided via a resistivedivider coupled to the source voltage. In addition, source selector mux713 also provides for selecting externally generated reference voltages,atest 736 and atest_n 746 that can be bonded to input pads of imagingdevice 100. By selecting one of the five reference voltages 706, 716,726, 736, 746, a known voltage can be applied to one or more of blocks719, 723, 729, 733, and output via ADC 134. The output of ADC 134 canthen be compared against a range of values to determine if imagingdevice 100 is within acceptable design limits.

In some embodiments, an additional test mechanism provides for digitaltesting of each column within pixel array 700. In such a test, pixeloutputs from pixel array 700 are paired with an output from reset level743. Reset level 743 consists of a group of pixels kept under a resetstate and a column amplifier identical to column amplifier 703. BISTpixels within pixel array 700 are selected and the outputs from thepixels are passed through a selected portion of column amplifier 703 andanalog mux 709. One side of the output is then paired with a known resetlevel 743 signal and selected via source selector mux 713. This pairingbetween the known reset level 743 and the output from the various BISTpixels is then passed through one or more of blocks 719, 723, 729, 733,and 134. The output of ADC 134 can then be compared against a range ofvalues to determine if imaging device 100 is within acceptable designlimits. Such a test provides an ability to test the operability of eachcolumn amplifier within column amplifier 703, using a digital tester andwithout requiring illumination of imaging device 100. The various BISTmodes for one particular embodiment are described in Table 4 below.

TABLE 4 Test Modes For Analog Blocks Negative Positive output of outputof source source selector selector mux 713 mux 713 im_test_en[3:0]im_test active connected to connected to Comments 0000 0 Positive outputof Negative output Normal analog mux 709 of analog mux operation with709 CDS 0001 1 Reset level 743 Negative output Normal of analog muxoperation 709 without CDS 0010 2 Reset level 743 Positive output Testingreset of analog mux sampling 709 branch of column amplifier 0011 3 atest736 atest_n 746 Testing with external signal 0100 4 vref_n 726 vref 706  −1 V 0101 5 common 716 vref 706 −0.5 V 0110 6 common 716 common 716    0 V 0111 7 common 716 vref_n 726   0.5 V 1000 8 vref 706 vref_n 726    1 V

In one particular embodiment, ADC 134 is a ten-bit ADC with pipelinearchitecture. The ten-bit output from ADC 134 is passed to translationunit 150. For the various dark-current calculations, four ten-bitoutputs are averaged, and the most significant ten-bits of the averagevalue are used.

Translation Unit

Translation unit 150 provides color processing for imaging device 100.Parameter and mode registers associated with translation unit 150 areloaded via control bus 101. A Bayer image is received from imaging unit130, processed by translation unit 150, and transferred to parallelinterface 190 for output via parallel interface I/O 192. Whileprocessing the image, statistics are calculated by translation unit 150.After completing the processing of an image, translation unit 150produces an interrupt to control unit 110 indicating that processing iscomplete and that the statistics are ready to be read by control unit110 via control bus 101.

Translation unit 150 receives the Bayer grid as input from imaging unit130. For each received pixel, two color values are missing and need tobe reconstructed. Translation unit 150 reconstructs the RGB set for eachpixel, performs various correction algorithms, and generates a YUVrepresentation of the image.

FIG. 24 provides a block diagram 1100 illustrating translation unit 150in greater detail. As illustrated, translation unit 150 includes an RGBreconstruction block 1110 (for demosaicing), a CIF-to-QCIF reductionblock 1130, a color and white-balance correction block 1140, a gammalook-up table (“LUT”) 1150, an RGB to YUV generation block 1160, atwenty-four bit YUV to 4:2:2 conversion block 1170, and a statisticsblock 1195. Statistics block 1195 includes a white balance statisticsblock 1180, an exposure statistics block 1185, and a G line-sum block1190 for providing a sum of G Bayer components per line used to detectflicker.

RGB reconstruction block 1110 receives the Bayer input from imaging unit130 and calculates all three color components for each pixel. MedianInterpolation method is selected for the Bayer-to-RGB conversion. Forgreen reconstruction from a pixel whose Bayer component is either red orblue, the green color component is given by the median value of the foursurrounding Green Bayer components. The median function is defined asthe average between the two values that remain when a series of fourvalues is evaluated, and the minimum and maximum values are removed.

For Blue reconstruction from a pixel whose Bayer component is either Redor Green, the Blue color component is given by the median value of thefour surrounding Blue Bayer components. For a pixel whose Bayercomponent is green located at a blue line, the blue value is the averageof the two adjacent blue Bayer components in the same line. Further, fora pixel whose Bayer component is green located at a red line, the bluevalue is the average of the two adjacent blue Bayer components in thesame column.

For red reconstruction at a pixel whose Bayer component is blue, the redcolor component is given by the median value of the four surrounding redBayer components. At a pixel whose Bayer component is green located at ared line, the red value is the average of the two adjacent red Bayercomponents in the same line. At a pixel whose Bayer component is greenlocated at a blue line, the red value is the average of the two adjacentred Bayer components in the same column. After reconstruction, theresulting RGB image is smaller than the Bayer input image by two columnshorizontally, and two lines vertically.

In some embodiments, imaging device 100 provides three methods forproducing a QCIF size image. In the first method, a QCIF sized window isread from a CIF sized array. For the method, no special processing isneeded and the RGB per pixel is calculated in the same manner as for aCIF sized image. The method provides a limited Field of View (“FOV”),but provides a good resolution.

In the second method, the array read out of pixel array 700 isdown-sampled. This method is implemented in the imaging unit 130. Thethird method is a Median Billinear method. In the method, the CIF sizedBayer image is first converted to an RGB image using the Median method.Then, if down-sampling is selected, the CIF image is reduced to a QCIFsize image using a (¼, ½, ¼) filter—horizontally and then vertically.

In some embodiments of imaging device 100, color correction block 1140can be used to compensate for any disparity between the RGB filters onimaging device 100 and a display connected via parallel interface I/O192. In some instances, sensor 132 is not equally sensitive to each ofthe color components. To compensate for the different sensitivity, andto emulate the human eye adjustment to different lighting conditions,the captured image undergoes a white-balance correction.

The coefficients of the color correction are combined into a singlematrix, and the corrected R′G′B′ values per pixel are calculatedaccording to the following equations:R′=CR1*R+CR2*G+CR3*B+RoffsetG′=CG1*R+CG2*G+CG3*B+GoffsetB′=CB1*R+CB2*G+CB3*B+Boffset

In addition, Gamma Correction LUT 1150 is used to gamma correct thevarious pixels. In one embodiment, Gamma correction LUT 1150 includes512 entries, where each entry is a nine-bit value. The data for Gammacorrection LUT 1150 is loaded to RAM within imaging device 100.

Gamma correction LUT 1150 receives a color component value from colorcorrection block 1140, and uses the value as an address to access theLUT stored in RAM. The value obtained from the LUT at the addressedlocation is stored until all three color components are received. Then,the three color components are passed together to RGB-to-YUV block 1160.

In YUV to 4:2:2 conversion block 1170, an input of an array of pixels,each with its own R, G and B values is reconstructed into an image ofYUV 4:2:2 values. The result is then passed to parallel interface 190.The process involves an interim conversion to a Y-Cr-Cb (4:4:4) format.The Y-Cr-Cb (4:4:4) is calculated as follows: R′G′B′ are given at an8-bit accuracy, in the range of 0-255, Y is limited to the range of16-235, and Cr and Cb are limited to the range of 16-240. Then, thefollowing equations are used to complete the conversion:Y=0.257R′+0.504G′+0.098B′+16Cb=−0.148R′−0.291G′+0.439B′+128Cr=0.439R′−0.368G′−0.071B′+128

The coefficients are loaded in {s,0.8} format, eight-bits of a fractionvalue, without integer bits, but with a single sign bit. In the samemanner, YUV can be calculated, without the clamping. The multiplycoefficients are programmable. The offset for the luminance can beenabled or disabled. When offset is enabled, the value of sixteen isused as offset for the luminance. The chrominance offset is fixed atone-hundred, twenty eight.

The twenty-four bit YUV value per each pixel (Y-Cr-Cb 4:4:4), isdown-sampled to a YUV 4:2:2 format. To do this, the Y value for eachpixel is retained, but each pair of consecutive luminance values (Y_(2n)and _(Y2n+1), n=0,1,2 . . . ) share a pair of Cb and Cr values. Theplacement of the Chrominance pair is on the even numbered pixel. FIG. 25illustrates this down-sampling process.

The color components of the odd number pixels are zeroed out, while thecolor components of the even numbered pixels are calculated as follows:

For first pixel in line:Cx′[0]=Cx[0](x is either r or b)

For subsequent pixels in the line:For n=1 to n, n is equal to (N/2−1), N is the line lengthtmp[2n]=(Cx[2n−1])+2*(Cx[2n])+(Cx[2n+1])(x is either r or b)

The result is rounded up and divided by 4:Cx′[2n]=¼(tmp[2n]+2) (x is either r or b)

Table 14 provides a summary of calculations performed by translationunit 150 as previously discussed.

TABLE 13 Calculation Summary Var Formula Accur. Remarks Input from C Care the Bayer grid outputs 10 bits unsigned imaging unit 130 Faulty C1C1 = C if not faulty, 10 bits unsigned correction replacement value iffaulty RGB per pel C2 C2 = (C1 + C1 + 1)/2 10 bits Average and round upC10 C10 = C2 or C1 10 bits unsigned Color C20 Color correctioncoefficient  8 bits + sign 8 bits value, 1 bit sign correction C21 C21 =C20 * C10 19 bits 18 bits magnitude, 1 bit sign [1, 18], C22 C22 =C21r + C21b + C21g 21 bits 21 bits 2's comp [1, 20] Toff Truncationparameter  4 bits Values 0 to 10 C23 C23 = truncated C22 21 bitsProgrammable C23 = {sign-ext, C22[20:t]} truncation [t − 1:0] lsbitstruncated, Coff Offset for Color correction  8 bits + sign 8 bits value,1 bit sign C24 C24 = (C23 + Coff * 2 + 1)/2 21 bits offset, round up(add 1) ignore lsbit, limit. Lower limit 0 or 16 Loadable Upper limit(0, 511 for color correction) C25 C25 = limited C24  9 bits select bits[10:1] Gamma C30 C30 = LUT[C25]  9 bits correction RGB to Y20 Formatconversion  8 bits + sign 8 bits value, 1 bit sign YUV coefficient Y21Y21 = Y20 * C30 19 bits 18 bits magnitude, 1 bit sign [1, 18], Y22 Y22 =Y21r + Y21b + Y21g 21 bits 21 bits 2's comp [1, 20] Toff Truncationparameter  4 bits Values 0 to 10 (fixed to 8) Y23 Y23 = truncated Y22 21bits Programmable Y23 = {sign-ext, truncation Y22[20:8]} 8 lsbitstruncated Yoff Offset for conversion  8 bits + sign 8 bits value, 1 bitsign Y24 Y24 = (Y23 + Yoff * 2 + 1)/2 21 bits offset, round up (add 1)ignore lsbit, limit. Lower limit 0 or 16 Loadable Upper limit 235/255for Y 240/255 for U/V Y25 Y25 = limited Y24  8 bits select bits [9:1]4:4:4 Yout Yout = Y25  8 bits to Ctmp Ctmp = (Yn − 1 + Yn * 2 + Yn + 1)10 bits 4:2:2 Cout Cout = (Ctmp[9:1] + 1)/2  8 bits

In addition, translation unit 150 collects a number of statistics duringthe previously discussed translations and calculations. At the end ofprocessing of a frame data, the resulting statistics are checked, andthe different registers that control the correction algorithms areupdated according to the results. The different statistics checked bythe various statistics blocks including white balance block 1180,exposure statistics block 1185, and G-line-sum block 1190. The variousstatistics checked can include, but are not limited to, white balance,exposure, minimum and maximum per color, and flicker.

Exposure statistics are determined by evaluating green Bayer pixels or Youtput data from translation unit 150. The evaluation results in aneight-bit histogram. There are eight seventeen-bit registers, eachholding the number of pixels whose Y or Bayer value falls within therange that is defined for that bin (ten-bit Bayer components are dividedby 4 before sorting). At the end of a frame processing, programmableprocessing core 112 reads the histogram results and calculates theexposure, offset and gain parameters for the following frames. Table 5lists the eight bins into which the values are assigned to create theeight bit histogram.

TABLE 5 Exposure Histogram Bins Bin Number Range Bin #0  0–31 Bin #132–63 Bin #2 64–95 Bin #3  96–127 Bin #4 128–159 Bin #5 160–191 Bin #6192–223 Bin #7 224–239

In addition, the statistics blocks of translation unit 150 detect theminimum and maximum value per each color, on the output of theBayer-to-RGB conversion block 1110. The results can be read fromregisters by programmable processing core 112 at the end of the frameprocessing.

Further, flicker detection can be performed using the statistics blocksof translation unit 150. Flickering can be caused by a mismatch betweenthe frequency of light impinging upon a detected image, and thefrequency at which the detected image is sampled. The effect offlickering is a modulation of the image information along the rows ofpixel array 700 (or down the columns, in some embodiments). One way toeliminate flickering, the frequency at which the detected image issample can be adjusted to coincide with the frequency of the lightimpinging upon the image. In one embodiment of the present invention, anestimate of the impinging light is calculated, and the sample frequencyis adjusted automatically. In some embodiments, to obtain an estimationof the impinging frequency with high confidence, a uniform image iscaptured, thus avoiding detection of modulation in the image itself.However, in other embodiments, any image is captured and flickering isautomatically detected and corrected.

FIG. 26 is a flow diagram of such a flicker correction system 2205 inaccordance with embodiments of the present invention. Flicker correctionsystem 2205 includes a feedback loop for automatically adjustingexposure to reduce flicker. Flicker correction system 2205 includescapturing an image (block 2206) and outputting the captured image. As isevident from the preceding discussion, the image is captured by sensor132 and provided to translation unit 150 via ADC 134. The captured imagereceived from ADC 134 is processed to determine if flickering is evident(block 2207). Then, the exposure is adjusted to account for any detectedflickering (block 2208). Using the adjusted exposure parameters, theimage is again captured and the adjustment process performed again. Insome embodiments, this process can be completed for one or more framesof the image. This use of additional image information provides for moreaccurate detection and correction. Further, such an iterative approachallows for continuous monitoring of flicker conditions. In otherembodiments, however, flicker detection and correction is onlyinfrequently performed, rather than continuously. In addition, in someembodiments, the process may adjust exposure duration in order toemphasize flicker artifacts and thereby increase the probability ofcorrect detection and/or reduce estimation time. Such an approach caninclude a manual adjustment whereby a user adjusts the capture frequencyuntil the user detects a noticeable flicker. After that, the flicker isautomatically detected and corrected. Alternatively, the adjustmentcreating the noticeable flicker can be automatically performed, andsubsequently corrected.

Alternatively, a fully passive mode of operation is also possible. Insuch a fully passive mode, exposure time is not altered by flickerdetection mechanism until the estimation is completed. Further, suchadjustments to the exposure time can be limited to multiples of adetected flicker period. In passive mode of operation the exposureadjustments can be introduced by other parts of the system including,for example, exposure selection sections that provide enough informationfor successful estimation of flicker frequency

A light source, I(t), illuminating an image can have a flickeringcomponent at a frequency f that can be modeled the following way:I(t)=a·cos(2πft+φ)+b,where a, b and φ are constants. R(n,m) is defined to be a imagereflectance function that corresponds to a pixel at location (n,m) ofpixel array 700. Using these definitions, the light impinging upon pixel(n,m) is given by:

$\begin{matrix}{{S\left( {n,m,t} \right)} = {{R\left( {n,m} \right)} \cdot {I(t)}}} \\{= {{b \cdot {R\left( {n,m} \right)}} + {a \cdot {R\left( {n,m} \right)} \cdot {{\cos\left( {{2\pi\; f\; t} + \varphi} \right)}.}}}}\end{matrix}$Thus, when an image is captured with exposure duration T, the value ofthe pixel (n,m) will be proportional to the following value:

$\begin{matrix}{{S\left( {n,m} \right)} = {\int_{n\;\Delta}^{{n\;\Delta} + T}{{S\left( {n,m,t} \right)}{\mathbb{d}t}}}} \\{{= {{R\left( {n,m} \right)} \cdot {\int_{n\;\Delta}^{{n\;\Delta} + T}{{I(t)}{\mathbb{d}t}}}}},}\end{matrix}$where Δ is the time period between sampling of two adjacent rows inpixel array 700. The integration of the incident light produces:

$\begin{matrix}\begin{matrix}{{\int_{n\;\Delta}^{{n\;\Delta} + T}{{I(t)}{\mathbb{d}t}}} = {{b \cdot T} + {\frac{a}{2\pi\; f}\left\lbrack {{\sin\left( {{2\pi\;{f\left( {{n\;\Delta} + T} \right)}} + \varphi} \right)} - {\sin\left( {{2\pi\;{fn}\;\Delta} + \varphi} \right)}} \right\rbrack}}} \\{= {{b \cdot T} + {\left\lbrack {\frac{a}{\pi\; f}{\sin\left( {\pi\;{fT}} \right)}} \right\rbrack \cdot {\cos\left( {{2\pi\; f\;\Delta\; n} + {\pi\;{fT}} + \frac{\varphi}{2}} \right)}}}} \\{{= {{b \cdot T} + {a^{\prime} \cdot {\cos\left( {{2\pi\; f^{\prime}n} + \varphi^{\prime}} \right)}}}},,}\end{matrix} \\{where} \\{f^{\prime} = {f \cdot \Delta}} \\{a^{\prime} \equiv {\frac{a}{\pi\; f}{\sin\left( {\pi\;{fT}} \right)}}} \\{\varphi^{\prime} \equiv {{\pi\;{fT}} + {\frac{\varphi}{2}.}}}\end{matrix}$Substituting this result provides:S(n,m)=R(n,m)·b·T+R(n,m)·a′·cos(2πf′n+φ′).

Thus, the resultant signal from pixel (n,m) consists of two components.The first component, R(n,m)·b·T, does not cause flickering artifacts;whereas, the second component can change intensity across rows of pixelarray 700. However, as illustrated by the preceding equation, when a′equals zero, no flickering exists. Therefore, since a′ is a function ofexposure duration, it is possible to select such exposure duration thata′ will vanish. Said another way, in some embodiments of the presentinvention, the flicker frequency is detected, and based on the detectedfrequency, the exposure time of pixel array 700 is adjusted to reduceflicker. To do this, T is selected to satisfy the following constraint:f·T=k,k=. . . ,−2,−1,0,1,2.

To detect the frequency components of the S(n,m), columns of each roware summed up to produce row averaged image S(n), where S(n) isdescribed by the following equation:

$\begin{matrix}{{S(n)} = {\sum\limits_{m = 0}^{M - 1}{S\left( {n,m} \right)}}} \\{= {\left( {{b \cdot T} + {a^{\prime} \cdot {\cos\left( {{2\pi\; f^{\prime}n} + \varphi^{\prime}} \right)}}} \right){\sum\limits_{m = 0}^{M - 1}{R\left( {n,m} \right)}}}} \\{= {{{R(n)} \cdot b \cdot T} + {{R(n)} \cdot a^{\prime} \cdot {{\cos\left( {{2\pi\; f^{\prime}n} + \varphi^{\prime}} \right)}.}}}}\end{matrix}$

In some embodiments, such a summation is performed by a hardware summingcircuit within translation unit 150 where the green and/or any otherBayer components for each row of pixel array 700 are summed. At the endof each row, an eighteen-bit value is sent to memory 118 of control unit110 using a direct memory access via control bus 101. This same processis repeated for each row of pixel array 700, with each sum being storedto memory 118. Alternatively, in other embodiments, such summing can beaccomplished using programmable processor core 112.

In some embodiments of the present invention, all rows and pixel valuestherein are summed. In other embodiments, only a subset of rowsincluding all pixel values therein are summed, while in yet otherembodiments, all rows, but only a subset of pixel therein are summed. Inyet another embodiment, a subset of the rows and a subset of the pixelstherein are summed.

From these sum values, S(n) is spectrally analyzed using softwareexecuted by programmable processing core 112 to detect the presence ofenergy corresponding to potential flickering frequency. FIG. 27illustrates a graph 2211 of a typical row average. More particularly,graph 2211 is a plot of the normalized row average 2212 verses the rownumber 2213 for an image detected by pixel array 700. Various spectralanalysis techniques can be used on the sum information to determine aflicker frequency, if any flicker is apparent. For purposes ofillustration, operation of flicker detection system 2205 is discussed inrelation to a Fourier analysis.

More particularly, two energy values, E₁ and E₂, are determined usingthe sum data. The two values are calculated as follows:

$\begin{matrix}{E_{i} = \frac{{{F\left\{ {S(n)} \right\}}}_{f = {f_{i}\Delta}}}{{{F\left\{ {S(n)} \right\}}}_{f = 0}}} \\{= \frac{{\sum\limits_{n = 0}^{N - 1}{{S(n)} \cdot {\mathbb{e}}^{{- 2}\pi\; f_{i}\Delta\; n}}}}{\sum\limits_{n = 0}^{N - 1}{S(n)}}} \\{= {\frac{{{\sum\limits_{n = 0}^{N - 1}{{S(n)} \cdot {\cos\left( {2\pi\; f_{i}\Delta\; n} \right)}}}} + {{\sum\limits_{n = 0}^{N - 1}{{S(n)} \cdot {\sin\left( {2\pi\; f_{i}\Delta\; n} \right)}}}}}{\sum\limits_{n = 0}^{N - 1}{S(n)}}.}}\end{matrix}$

In some embodiments, a simple comparison between the energy values E₁and E₂ is used to provide an estimation of the flicker frequency. Suchestimations, however, can prove unreliable when any ordinary image isthe source of the sum information rather than a controlled, constantimage. Thus, in various embodiments, a more reliable estimation of theflicker frequency is obtained by calculating a set of E₁ and E₂ pairsacross various frames detected by pixel array 700, and comparing thevariance of E₁ and E₂.

This approach provides a number of sampling points across various imagesand, thus, a more robust estimation of the flicker frequency.Improvement results because there is likely a different phase of theillumination for each frame, which provides various modulations of R(n).After calculation of E₁ and E₂, corresponding to two possible flickeringfrequencies, the energy of the present flickering will vary depending onthe phase; whereas the energy of the other frequency which depends onthe R(n) only, will remain almost the same for all frames. Therefore,the comparison between the standard deviation of E₁ and E₂ producesreliable results. This procedure is presented in more detail below basedon the following definition of variance:

${V_{i} = {{\frac{1}{K}{\sum\limits_{k = 0}^{K - 1}\left( E_{i}^{k} \right)^{2}}} - \left( {\frac{1}{K}{\sum\limits_{k = 0}^{K - 1}E_{i}^{k}}} \right)^{2}}},$where K is the number of frames that were captured and E_(i) ^(k) is theenergy value corresponding to frequency f_(i) of the k-th image. Toimprove the performance, frames can be captured at two or more differentexposures that each exhibit flickering. As an example, the followingequation describes exposure times creating such a dual flicker:

${T = {\left( {l + \frac{1}{2}} \right) \cdot \frac{1}{f_{i}}}},{i = 1},{2;{l = 0}},1,2,\ldots$When both V₁ and V₂ are below some predefined threshold, no flickeringhas been detected. Otherwise, the estimated flickering frequency is thefrequency that produced larger V_(i).

FIG. 28 illustrates a flow diagram 2241 of a method for detecting andcorrecting flicker in accordance with the present invention. Followingflow diagram 2241, a frame is captured from pixel array 700 (block2243). The lines of the frame are summed, and E₁ and E₂ are calculatedas previously described (block 2247). Next, it is determined if asufficient amount of information has been gathered to make a reliabledetermination of flicker frequency (block 2249). In one embodiment,satisfactory results were obtained when seventeen frames of data areobtained before determining the flicker frequency. If not, moreinformation is gathered by processing additional frames. Alternatively,where sufficient information has been gathered, V₁ and V₂ are calculatedusing the gathered information (block 2251). The determined variancevalue is set equal to the maximum of V₁ and V₂, and the frequency is setto the frequency f_(i) at which the maximum variance was detected (block2253).

Where the determined variance is less than a predefined threshold levelD (block 2257), it is determined that no flicker is detected (block2259), and the process is repeated to detect a change in conditions thatresults in flicker. Alternatively, where the determined variance isgreater than the a predefined threshold level D (block 2257), a flickerfrequency of f_(i) is detected (block 2261) and the exposure duration isconstrained to a multiple of 1/f_(i) (block 2263) to correct thedetected flicker.

In some embodiments, the previously described approach is simplified toreduce computational complexity. More particularly, instead ofcalculating the exact Fourier component, it is replaced by anapproximation of the energy value according to the following equation:

${{\hat{E}}_{i} = \frac{{{\sum\limits_{n = 0}^{N - 1}{{S(n)} \cdot {{SC}\left( {2\pi\; f_{i}\Delta\; n} \right)}}}} + {{\sum\limits_{n = 0}^{N - 1}{{S(n)} \cdot {{SS}\left( {2\pi\; f_{i}\Delta\; n} \right)}}}}}{\sum\limits_{n = 0}^{N - 1}{S(n)}}},{where}$${{SC}(x)} = {{{sign}\left( {\cos(x)} \right)} = \left\{ {{\begin{matrix}{1,} & {{- \frac{\pi}{2}} \leq x \leq \frac{\pi}{2}} \\{{- 1},} & {\frac{\pi}{2} \leq x \leq \frac{3\pi}{2}}\end{matrix}{and}{{SS}(x)}} = {{{sign}\left( {\sin(x)} \right)} = \left\{ \begin{matrix}{1,} & {0 \leq x \leq \pi} \\{{- 1},} & {\pi \leq x \leq {2{\pi.}}}\end{matrix} \right.}} \right.}$However, such a computational reduction can cause degradation infrequency estimation performance. Nevertheless, even after suchdegradation, satisfactory results of flicker detection and correctioncan be obtained. Moreover, in some embodiments, only one of thecomponents is used during the estimation process. Such an approachresults in insignificant performance degradation which may be overcomeby increasing number of considered frames K.

In yet other embodiments where greater confidence in the estimatedflickering frequency is desired, a more complex approach is utilized. Insuch approaches, various measures of confidence can be implemented toprevent frequency switches (block 2263) when an incorrect estimate offlicker frequency is detected. Further, in some embodiments, residualflickering (i.e. flickering due to slight variance around the flickeringfrequency) can be eliminated.

More particularly, one such improvement increases the probability of acorrect estimation by utilizing more pixel values in the variouscalculations. Instead of using just a standard deviation of the energythat corresponds to possible flickering frequency to make decision, onemay base the estimation on a number, L, of consequent standarddeviations. The following equations illustrate such an approach:

$W_{1} = {\sum\limits_{l = 0}^{L - 1}\frac{V_{1}^{l}}{V_{2}^{l}}}$$W_{2} = {\sum\limits_{l = 0}^{L - 1}\frac{V_{2}^{l}}{V_{1}^{l}}}$$C_{1} = {\sum\limits_{l = 0}^{L - 1}{I\left( {V_{1}^{l} > V_{2}^{l}} \right)}}$$C_{2} = {\sum\limits_{l = 0}^{L - 1}{I\left( {V_{2}^{l} > V_{1}^{l}} \right)}}$$M = {\max\limits_{l}\left\{ {{\sum\limits_{k = 0}^{K - 1}E_{1}^{k,l}} + E_{2}^{k,l}} \right\}}$where I(·) is an indicator function defined below:

${I({Condition})} = \left\{ \begin{matrix}{1,} & {{if}\mspace{14mu}{Condition}\mspace{14mu}{is}\mspace{14mu}{true}} \\{0,} & {{otherwise},}\end{matrix} \right.$and V_(x) ^(l), E_(x) ^(k,l) denotes the l-th standard deviation and theenergy of k-th frame in l-th run, respectively. Note that the totalnumber of frames that should be acquired prior the decision is K·L. Thedecision is made based on the following set of rules:

-   -   1. If M is below a predefined flickerless threshold D₀ no        flicker artifact was detected.    -   2. If W₁>D₁∩C₁>D₂, flicker at f₁ was detected.    -   3. If W₂>D₁∩C₂>D₂, flicker at f₂ was detected.    -   4. Otherwise, the information is ambiguous and no decision is        taken.        This method reduces drastically the probability of both        misdetection and incorrect detection. However, the delay and        probability of false alarm are increased, which can be less        critical in some applications.

Further, to prevent the situation when occasional wrong frequencyestimation changes the exposure to a value that leads to flickerartifact, some embodiments include additional mechanisms to reduce thispossibility. One such approach for avoiding this situation includes anadditional counter, referred to as a confidence counter. The confidencecounter is initialized to zero. Whenever flicker at a first frequency,f1, is estimated the confidence counter is increased by one for eachestimation until a limit, B, is reached. If estimated flicker frequencyis a second frequency, f2, the confidence counter is decreased by onefor each estimation until a limit, negative B, is reached. In a case ofambiguity, no change is made to the confidence counter. When theconfidence counter is positive, the global estimation is f1 and exposureis limited to multiple of 1/f1. When the confidence counter is negative,the global estimation is f2 and exposure is limited to multiple of 1/f2.If the confidence counter is zero previous estimation is used. Thisreduces the possibility of spurious data creating an unstable exposuretime.

In some embodiments, flicker artifacts may be removed electronically byvarying the gain of certain pixels to accommodate light intensityvariations, thereby creating an effective intensity that remainsconstant. Such embodiments are enabled by estimating flicker frequencyand phase and using these estimates to determine which pixel gains toadjust. The frequency may be estimated in previously-described ways.Both frequency and phase may be estimated in ways to be describedhereinafter.

The previously-derived formula:S(n,m)=R(n,m)·b·T+R(n,m)·a′·cos(2πf′n+φ′),may be rearranged as follows:S(n,m)=R(n,m)·[b·T+a′·cos(2πf′n+φ′)].In order to eliminate flicker, the illumination component may be madeconstant for every pixel in the image as follows:g(n)·[b·T+a′·cos(2πf′n+φ′)]=c,where c is a constant and g(n) is a compensating gain. The value of theconstant may be selected in any number of ways. In some embodiments, itis selected to be:c=b·T+a′.This selection preserves picture quality by making the compensating gainalways equal to or greater than unity, which prevents incorrectlysaturated pixels. Hence,

${{g(n)} = \frac{{b \cdot T} + a^{\prime}}{{b \cdot T} + {a^{\prime} \cdot {\cos\left( {{2\pi\; f^{\prime}n} + \varphi^{\prime}} \right)}}}},{and}$$\begin{matrix}{{{S\left( {n,m} \right)} = {{g(n)} \cdot {R\left( {n,m} \right)} \cdot \left\lbrack {{b \cdot T} + {a^{\prime} \cdot {\cos\left( {{2\pi\; f^{\prime}n} + \varphi^{\prime}} \right)}}} \right\rbrack}},} \\{{= {{R\left( {n,m} \right)} \cdot \left\lbrack {{b \cdot T} + a^{\prime}} \right\rbrack}},}\end{matrix}$where I′=b·T+a′ is the effective illumination intensity. In someembodiments, different compensating gains are used to preserve correctcolor appearance due to possible variations of illumination spectrumwith intensity.

Hence, as is apparent from the above formulas, eliminating flickerwithout constraining the exposure duration becomes a matter ofestimating the frequency and phase of the light source. This can beaccomplished in any of a number of ways, some of which were describedabove.

Having described electronic flicker elimination embodiments generally,attention is directed to FIG. 29, which illustrates an exemplary method2400 of electronically eliminating flicker. In this embodiment, flickeris eliminated or reduced “on the fly” by gain-adjusting the pixel valueas a function of estimated frequency and phase and the pixel's location.

The method 2400 begins at block 2402 at which location a gaincompensating circuit receives the pixel's value. At blocks 2404 and2406, the frequency and phase estimates are received by the gaincompensating circuit. At block 2408, a pixel's position information isreceived. Although shown as serial operations, this in not necessarilythe case. In some embodiments, the gain compensating circuitry receivesthe frequency estimate, the phase estimate, a pixel's value, and thepixel's position in parallel.

At block 2410, a gain, specific to the pixel, is calculated. The gain isthen applied to the pixel's value at block 2412 to determine the pixel'scorrected value.

The foregoing method 2400 may be implemented in hardware or software.Further, any of the previously-described methods for estimatingfrequency and/or phase may be used.

In some embodiments, the light source's frequency and phase offset aredetermined after an image is captured. Hence, the gain correctiontechnique described above may be accomplished as a post-processing stepthat may be performed off line. In such embodiment, an initial pixelvalue may be stored for each pixel of the image, and the pixel valuesare thereafter analyzed to determine the phase offset. The frequency maybe determined before the image is captured, as described previously, ormay be determined in post processing as well. Once the phase andfrequency are known, a compensating gain for each pixel may bedetermined and each pixel's value adjusted accordingly as describedabove.

The foregoing is merely exemplary of the many ways flicker may bereduced and/or eliminated. Those skilled in the art will appreciateother methods in light of the disclosure herein.

Parallel Interface

Parallel interface 190 produces the video output of imaging device 100.parallel interface 190 includes a ten-bit parallel data bus, a verticalframe signal, a clock, and a qualifying signal. Parallel interface 190can also produce a serial output selectable by programming a register.

When operating in a parallel mode, imaging device 100 outputs a videoimage in a parallel format. The parallel format includes a clock signalalong with data signals and qualifying signals each synchronized to theclock signal. In some embodiments of imaging device 100, the effectiveedge of the clock signal (i.e., rising or falling edge) is programmable.

The data signals include a ten-bit wide data bus carrying either theBayer grid data, or the YUV 4:2:2 output data, where the type of data isdetermined by monitoring a programmed register. The eight-bit YUV datais presented on the most significant data signals (i.e., bits threethrough ten of the ten-bit bus). On each effective edge of the clock, asingle data byte is transferred on the data bus where a horizontalqualifying signal may be asserted. The horizontal qualifying signal isasserted only during valid image data, or to qualify optionalstart-of-line and end-of-line markers.

A vertical qualifying signal may be asserted on the effective edge ofthe clock to mark the beginning of the first line in a new image frame.The duration of the vertical qualifying signal is programmable via aregister implemented as part of imaging unit 130. FIG. 30 illustratesthe vertical qualifying signal, VALIDV, and the horizontal qualifyingsignal, VALIDH, in relation to a pixel array where the active imageportion of the array is designated. As illustrated, VALIDV is a verticalsynchronization signal whose effective edge (programmable by setting aregister value) marks the beginning of the first line in a new imageframe.

FIG. 30 illustrates the signals in relation to data signals referred toa valid data, and FIGS. 35 and 36 illustrate a bitwise view of thesignals in relation to Bayer and YUV data, respectively. As illustrated,when sending out the data in Bayer-grid format, a “RED” line will alwaysbe the first line in the frame. When sending out the data in YUV outputformat, the order of the color components is programmable via a registervalue.

Control unit 110 controls the frame rate output via parallel interface190 via a C_outframe signal. Via the signal, control unit 110 enables ordisables the output of a full frame. When the signal is deasserted,VALIDH and VALIDV signals are stuck at an inactive state and the datasignals hold the background value throughout the disabled frame time. Bydeasserting C_outframe, a frame of data can be skipped as illustrated inFIG. 34. The type of data output via parallel interface 190 isprogrammable. When Bayer data is output, the data volume is an 8/9/10bit word.

If data markers are added, eight more bytes per line must be taken intoconsideration in the image buffer that is allocated for the application.There are two types of BAYER data lines: red lines and blue lines. Redlines carry green and red Bayer components. The green and red data bytesare interleaved where the even numbered bytes (0, 2 to 2n) are green,and the odd numbered bytes (1,3 to 2n+1) are red. On the other hand,blue lines carry blue and green Bayer components, where the componentsare interleaved with the even numbered bytes (0, 2 to 2n) being blue,and the odd numbered bytes (1,3 to 2n+1) being green. When sending Bayerdata out on parallel interface 190, a red line is the first line in theframe.

When YCrCb 4:2:2 data is output, the data volume is two bytes per pixeland no extra pixels are transmitted beyond the CIF/QCIF size. If datamarkers are added, eight more bytes per line must be taken intoconsideration in the image buffer that is allocated for the application.The data in the YCrCb 4:2:2 output format is ordered in severalconfigurations, according to the setting in a programmable colorregister. The following illustrates four possible color orders:

-   -   00—SOL, Y₀, U₀, Y₁, V₀, Y₂, U₂, Y₃, V₃, to Y_(n−2), U_(n−2),        Y_(n−1), V_(n−2), EOL    -   01—SOL, Y₀, V₀, Y₁, U₀, Y₂, V₂, Y₃, U₃, to Y_(n−2), V_(n−2),        Y_(n−1), U_(n−2), EOL    -   10—SOL, U₀, Y₀, V₀, Y₁, U₂, Y₂, V₃, Y₃, to U_(n−2), Y_(n−2),        V_(n−2), Y_(n−1), EOL    -   11—SOL, V₀, Y₀, U₀, Y₁, V₂, Y₂, U₃, Y₃, to V_(n−2), Y_(n−2),        U_(n−2), Y_(n−1), EOL

In some embodiments, extra data markers are inserted at the beginningand end of each image line. This insertion is optional and can beselected via a programmable register. The following are four examples ofdata marker types:

-   -   Start of First Line marker—This marker precedes the data of the        first line in a frame.    -   Start of Line Marker—This marker precedes the data of each line        in a frame, except for the first line. When Bayer output mode is        selected, bit #5 of the marker content differentiates between a        “red” line and a “blue” line (0 and 1 respectively).    -   End of Last Line Marker—This marker immediately follows the data        of the last line in a frame.    -   End of Line Marker—This marker immediately follows the data of        each line in a frame, except for the last line.

In addition to outputting data in a parallel mode, parallel interface190 can format the data for serial output. The Transmitted data size forserial output mode is eight bits per transmitted “word”. Bayer outputprovides eight bits per pixel, and YUV output is sixteen bits per pixel.The bytes are transmitted with the most significant bits first. During acontinuous transfer, the most significant bit of byte N immediatelyfollows the least significant bit of byte N−1.

In some embodiments, imaging device 100 is integrated into systems wherethe Host is pin limited. For example, in one particular embodiment, thehost is a Digital Signal Processor capable of receiving video via a twoor three pin interface. In such systems, imaging device 100 can beconfigured to provide video output via a serial interface. In someembodiments, the serial mode provided by imaging device 100 relies onelements consistent with ITU-RBT.656-4 standard. ITU-RBT.656-4 standardrecommendation is available from the International TelecommunicationUnion, the entirety of which is incorporated herein by reference for allpurposes. In one particular embodiment, imaging device 100 can beconfigured to operate in different serial output modes: a clock-qualifymode and a synchronous serial interface (“SSI”) mode. Such functionalityprovides an ability to function in low cost, reduced pin-countenvironments.

In contrast to ITU-RBT.656-4 standard, in embodiments of the presentinvention where data markers are used, the EAV and SAV markers aredifferent. In particular, the second most significant bit (i.e. bit six)of the markers is a special frame marker used to indicate the start ofthe first line of a frame and the end of the last line of a frame.Further, bit five indicates the Bayer color, where a logical oneindicates blue and a logical 0 indicates red.

In the serial output modes, either two or three pins of the thirteen-bitparallel video interface 192 are utilized to output the serial videodata. Further, in each of the serial output modes, the frequency of theclock, DSCLK 2920 (FIG. 35), used to output the serial data is eighttimes (“8×”) that of a clock, CLK, used to output parallel video datavia parallel video interface 192. In some embodiments, DSCLK 2920 isprovided via an external pin to imaging device 100. In such cases, theinput clock source is internally divided to provide clocking useful foroperating parallel video interface 192 when both parallel and serialvideo data are being provided from imaging device 100. In otherembodiments, an internal phase lock loop is used to generate the variousoutput clocks for use in relation to both the serial and parallel outputmodes of imaging device 100.

In operation, parallel video data is received from translation unit 150.Where a serial output mode is selected via a programmable register, theparallel data is shifted out via the designated serial pins of parallelvideo interface 192 using the 8× clock. This shifting occurs bytransmitting the most significant bit of any parallel data first andfollowing in succession with bits of decreasing significance.

Clock-qualify mode is a two wire interface with clock and output datasignals, DSCLK 2920 and DSDATA 2940. Imaging device 100 acts as a masterof the interface. As such, imaging device 100 activates DSCLK 2920 whenthere is valid data to be sent out via DSDATA 2940. FIG. 35 illustratesthe clock-qualify mode where data can be output at two rates representedby DSCLK/2 2950 and DSCLK 2920, both being synchronized to CLK ofimaging device 100. As illustrated, DSDATA 2940 is provided by imagingdevice 100 on the rising edge of DSCLK/2 2950 and latched by the host onthe falling edge of DSCLK/2 2950. It should be recognized that twice thetransfer rate is possible where the same approach is used in relation toDSCLK 2920. Using the approach, eight bits are transferred from the mostsignificant bit to the least significant bit. In some embodiments, onedeasserted cycle of DSCLK 2920 or DSCLK/2 2950 indicates a wordboundary, while in other embodiments, word boundaries are contiguousallowing for a continuous transfer of data based on a continuing periodof DSCLK 2920 or DSCLK/2 2950.

SSI mode is a three wire interface, with clock (DSCLK 2920), output data(DSDATA 2940) and frame signals (DSFRAME 2930). Imaging device 100 actsas a master of the interface, producing all three signals. FIG. 36illustrates the operation of SSI mode with DSCLK 2920, DSFRAME 2930, andDSDATA 2940. As illustrated, DSFRAME 2930 is asserted by imaging device100 prior to a rising edge of DSCLK 2920. The host accepts the assertedDSFRAME 2930 on a rising edge 2910 of DSCLK 2920. On the eight followingrising edges of DSCLK 2920, one of the eight bits of a byte transfer areprovided by imaging device 100 as DSDATA 2940 to a receiving host. Thebits are provided with the most significant bit of the by first followedsequentially by bits of decreasing significance.

Testability Unit

Imaging device 100 can additionally include various mechanisms forperforming tests of the device. Using one such mechanism, abuilt-in-self-test (BIST) is implemented to facilitate productiontesting of ADC 134 and the various DAC circuitry. The BIST circuitcreates a sequence of values that are passed to ADC 134 afterapplication of the gain and two offset parameters as previouslydiscussed. The output from sensor 132. With this value presented to ADC134, the outputs of ADC 134 is checked against a known output. If theoutput falls within a programmable tolerance level, imaging device 100is acceptable. If not, imaging device 100 is rejected.

In addition, a BIST mechanism for sensor 132 is implemented. A minimumand maximum value is defined for each color within sensor 132. Sensor132 is illuminated by a monochrome color, and the resulting image istested. The value of each pixel is checked. At the end of the frame,three values can be read per each color. The number of pixels below theminimum value, the number of pixels above the maximum value, and thenumber of pixels falling within the minimum and maximum values isdetermined. Additionally, BIST mechanisms are implemented to testvarious RAMs and registers within imaging device 100 and to return passor fail results.

The invention has now been described in detail for purposes of clarityand understanding. However, it will be appreciated that certain changesand modifications may be practiced within the scope of the appendedclaims. Accordingly, it should be recognized that many other systems,functions, methods, and combinations thereof are possible in accordancewith the present invention. Thus, although the invention is describedwith reference to specific embodiments and figures thereof, theembodiments and figures are merely illustrative, and not limiting of theinvention. Rather, the scope of the invention is to be determined solelyby the appended claims.

1. A method of capturing an image of a scene, the method comprising:providing an imaging device having an array of pixels, wherein eachpixel is configured to provide a pixel output; determining a frequencyof light produced by a light source illuminating the scene; capturing animage of the scene using the pixel array, wherein a first capture timefor one or more pixels in the array is different from a second capturetime of one or more other pixels in the array; and for each pixel in thearray: determining a relative phase of the light with respect to thecapture time of the pixel; determining a compensating gain based on therelative phase of the light and the frequency of the light; andadjusting the pixel output of the pixel in relation to the compensatinggain; wherein determining a frequency of light produced by a lightsource illuminating the scene comprises: capturing an image of agenerally uniform scene; and comparing pixel outputs relating to thecaptured image to determine a variation between pixel outputs for one ormore pixels in the array of pixels.
 2. The method of claim 1, whereinthe imaging device comprises a CMOS imaging device.
 3. The method ofclaim 1, wherein determining a compensating gain based on the relativephase of the light and the frequency of the light further includesdetermining a compensating gain based on a color associated with thepixel.
 4. A method of capturing an image of a scene, the methodcomprising: providing an imaging device having an array of pixels,wherein each pixel is configured to provide a pixel output; determininga frequency of light produced by a light source illuminating the scene;capturing an image of the scene using the pixel array, wherein a firstcapture time for one or more pixels in the array is different from asecond capture time of one or more other pixels in the array; and foreach pixel in the array: determining a relative phase of the light withrespect to the capture time of the pixel; determining a compensatinggain based on the relative phase of the light and the frequency of thelight; and adjusting the pixel output of the pixel in relation to thecompensating gain; wherein determining a frequency of light produced bya light source illuminating the scene comprises: using pixel outputsrelating to the captured image to calculate a sum of pixel outputs for aplurality of pixels in a row of pixels; spectrally analyzing the sum;calculating at least two energy values associated with the sum; andcomparing the at least two energy values.
 5. An image capture device,comprising: an array of pixels, wherein each pixel is configured toprovide a pixel output; and a programmable core, wherein theprogrammable core includes instructions executable by the programmablecore to: determine a frequency of light produced by a light sourceilluminating a scene; cause the pixel array to capture an image of thescene, wherein a first capture time for one or more pixels in the arrayis different from a second capture time of one or more other pixels inthe array; and for each pixel in the array: determine a relative phaseof the light with respect to the capture time of the pixel; determine acompensating gain based on the relative phase of the light and thefrequency of the light; and adjust the pixel output of the pixel inrelation to the compensating gain; wherein in programming the core todetermine a frequency of light produced by a light source illuminatingthe scene, the instructions program the core to: capture an image of agenerally uniform scene; and compare pixel outputs relating to thecaptured image to determine a variation between pixel outputs for one ormore pixels in the array of pixels.
 6. The image capture device of claim5, wherein the image capture device comprises a CMOS image capturedevice.
 7. The image capture device of claim 5, wherein in programmingthe core to determining a compensating gain based on the relative phaseof the light and the frequency of the light, the instructions programthe core to determine a compensating gain based on a color associatedwith the pixel.
 8. An image capture device, comprising: an array ofpixels, wherein each pixel is configured to provide a pixel output; anda programmable core, wherein the programmable core includes instructionsexecutable by the programmable core to: determine a frequency of lightproduced by a light source illuminating a scene; cause the pixel arrayto capture an image of the scene, wherein a first capture time for oneor more pixels in the array is different from a second capture time ofone or more other pixels in the array; and for each pixel in the array:determine a relative phase of the light with respect to the capture timeof the pixel; determine a compensating gain based on the relative phaseof the light and the frequency of the light; and adjust the pixel outputof the pixel in relation to the compensating gain wherein in programmingthe core to determine a frequency of light produced by a light sourceilluminating the scene, the instructions program the core to: use pixeloutputs relating to a captured image to calculate a sum of pixel outputsfor a plurality of pixels in a row of pixels; spectrally analyze thesum; calculate at least two energy values associated with the sum; andcompare the at least two energy values.
 9. An image capture device,comprising: an array of pixels, wherein each pixel is configured toprovide a pixel output; means for determining a frequency of lightproduced by a light source illuminating a scene; means for determining aphase of the light relative to a pixel capture time; means fordetermining a compensating gain for each pixel in the array based on thefrequency and the phase; and means for adjusting each pixel's pixeloutput, during an image capture, in relation to the pixel's compensatinggain; wherein the means for determining a frequency of light produced bya light source illuminating the scene comprises: means for capturing animage of a generally uniform scene; and means for comparing pixeloutputs relating to the captured image to determine a variation betweenpixel outputs for one or more pixels in the array of pixels.
 10. Theimage capture device of claim 9, wherein the image capture devicecomprises a CMOS image capture device.
 11. The image capture device ofclaim 9, wherein each pixel of the array includes a plurality ofsub-pixels and wherein the image capture device further includes meansfor determining a compensating gain for each sub-pixel of each pixel.12. An image capture device, comprising: an array of pixels, whereineach pixel is configured to provide a pixel output; means fordetermining a frequency of light produced by a light source illuminatinga scene; means for determining a phase of the light relative to a pixelcapture time; means for determining a compensating gain for each pixelin the array based on the frequency and the phase; and means foradjusting each pixel's pixel output, during an image capture, inrelation to the pixel's compensating gain; wherein the means fordetermining a frequency of light produced by a light source illuminatingthe scene comprises: means for using pixel outputs relating to acaptured image to calculate a sum of pixel outputs for a plurality ofpixels in a row of pixels; means for spectrally analyzing the sum; meansfor calculating at least two energy values associated with the sum; andmeans for comparing the at least two energy values.